Download file via http

[Kai Meyer]

On 12/13/2011 11:10 AM, Regan Heath wrote:
> On Tue, 13 Dec 2011 17:58:57 -0000, Kai Meyer <kai at unixlords.com> wrote:
>
>> On 12/13/2011 10:39 AM, Vladimir Panteleev wrote:
>>> On Tuesday, 13 December 2011 at 17:29:20 UTC, Kai Meyer wrote:
>>>> I get bytes_needed from the Content-Length header. The I get the
>>>> correct number of bytes from the Content-Length, bytes_needed gets the
>>>> right value, but the resulting file isn't right. The file has the
>>>> right number of bytes, but I appear to have an extra '0a' at the very
>>>> beginning of the file, but if I do 'ss.getchar()', to get rid of it, I
>>>> get an exception that there's not enough data in the stream.
>>>
>>> In an HTTP request, the headers are separated from the body by an empty
>>> line. Headers use CR/LF line endings, so the body is always preceded by
>>> a 0D 0A 0D 0A sequence. It looks like your code is not snipping the last
>>> 0A.
>>>
>>> Where did the getchar method come from? There is no mention of it in
>>> Phobos. Perhaps you could try the read(out ubyte) method?
>>
>> http://www.d-programming-language.org/phobos/std_stream.html
>>
>> Oh, I meant getc(), not getchar(), sorry. It looks like read(out
>> ubyte) worked on windows.
>>
>> I'm using ss.readLine() to pull headers from the stream. When the
>> string returned from ss.readLine() is empty, then I move on to the
>> stream. I'm going to be using this application on Windows, Linux, and
>> Mac, which is why I chose D. This feels like I've just entered the
>> newline/carriage return nightmare. Should I not be using readLine()?
>> Or is there some generic code that will always work and stick me at
>> the beginning of the file?
>
> I would have expected what you're doing to work.
>
> IIRC when you make a GET request you send HTTP/1.0 or HTTP/1.1 or
> similar in the GET request line, right? (my memory of the syntax is a
> bit fuzzy).
>
> Warning, wacky idea with little/no backing knowledge.. IIRC using
> HTTP/1.1 introduced additional data into the response, lengths or
> checksums, or something - I never did get to the bottom of it. But, if
> you change to using HTTP/1.0 they go away. I wonder if the 0A is related
> to that. As a simple test you could try HTTP/1.0 in your request and
> look at the response content-length, it might just be 1 byte shorter as
> a result.
>
> Regan
>

Doing a read(out ubyte) read a single byte from the stream, and allowed 
me to continue to read the full content-length number of bytes. 
Switching read(out ubyte) for a simple getc() caused the 
not-enough-bytes in stream exception. I'm now downloading the correct 
size file with bytes in the correct places after calling read().

I may or may not play with HTTP/1.0. I need to turn my attention to 
other matters at the moment though, since it currently "works".

-Kai Meyer