(Phobos - SocketStream) Am I doing something wrong or is this

[Travis Boucher]

Zane wrote:
> Travis Boucher Wrote:
> 
>> There are a few issues with your implementation.
>>
>> First, parse the headers properly.  Below see my trivial implementation. 
>>   You want to parse them properly so you can find the correct 
>> end-of-headers, and check the size of the content from the headers.
>>
>> readLine() looks to be designed for a text based protocol.  The biggest 
>> issue is with the end-of-line detection.  "\r", "\n" and "\r\n" are all 
>> valid end-of-line combinations and it doesn't seem to do the detection 
>> in a greedy manor.  This leaves us with a trailing '\n' at the end of 
>> the headers.
>>
>> The implementation of readBlock() doesn't seem to really wait to fill 
>> the buffer.  It fills the buffer, if it can.  This is pretty standard
>> of a read on a socket.  So wrap it in a loop and read chunks.  You want 
>> to do it this way anyway for many reasons.  The implementation below 
>> double-buffers which does result in an extra copy.  Although logically 
>> this seems like a pointless copy, but in a real application it is very 
>> useful many reasons.
>>
>> Below is a working version (but still has its own issues).
>>
>> #!/usr/bin/gdmd -run
>>
>> import std.stream;
>> import std.stdio;
>> import std.socket;
>> import std.socketstream;
>>
>> import std.string;		// for header parsing
>> import std.conv;		// for toInt
>>
>> import std.c.time;
>>
>> int main()
>> {
>> 	char[] line;
>> 	ubyte[] data;
>> 	uint num = 0;
>>
>> 	TcpSocket socket = new TcpSocket(new InternetAddress("www.google.com", 
>> 80));
>>
>> 	socket.send("GET /logos/bigbird-hp.gif HTTP/1.0\r\n\r\n");
>>
>> 	SocketStream socketStream = new SocketStream(socket);
>> 	
>> 	string[] response;	// Holds the lines in the response
>> 	while(!socketStream.eof)
>> 	{
>> 		line = socketStream.readLine();
>>
>> 		if (line=="")
>> 			break;
>>
>> 		// Append this line to array of response lines
>> 		response ~= line;
>> 	}
>>
>> 	// Due to how readLine() works, we might end up with a
>> 	//trailing \n, so
>> 	// get rid of it if we do.
>> 	ubyte ncr;
>> 	socketStream.read(ncr);
>> 	if (ncr != '\n')
>> 		data ~= ncr;
>>
>>
>> 	// D's builtin associative arrays (safe & easy hashtables!)
>> 	string[char[]] headers;	
>> 	
>> 	// Parse the HTTP response.  NOTE: This is a REALLY bad HTTP
>> 	// parser. a real parser would handle header parsing properly.
>> 	// See RFC2616 for proper rules.
>> 	foreach (v; response) {
>> 		// There is likely a better way to do this then
>> 		// a join(split())
>> 		string[] kv_pair = split(v, ": ");
>> 		headers[tolower(kv_pair[0])] = join(kv_pair[1 .. $], ":");
>> 	}
>>
>> 	foreach (k, v; headers)
>> 		writefln("[%s] [%s]", k, v);
>>
>> 	uint size;
>> 	if (isNumeric(headers["content-length"])) {
>> 		size = toInt(headers["content-length"]);
>> 	} else {
>> 		writefln("Unable to parse content length of '%s' to a number.",
>> 			headers["content-length"]);
>> 		return 0;
>> 	}
>> 	// This fully buffers the data, if you are fetching large files you
>> 	// process them in chunks rather then in a big buffer.  Also, this
>> 	// does not handle chunked encoding, see RFC2616 for details.
>> 	while (data.length < size && !socketStream.eof) {
>> 		ubyte[4096] buffer;
>> 		num = socketStream.readBlock(buffer.ptr, 4096); // read 4k at a time
>> 		writefln("Read %s bytes [%s/%s] (%s%%)",
>> 			num, data.length, size, (cast(float)data.length/cast(float)size)*100);
>>
>> 		// Process the buffer, in this case just copy it to the data
>> 		// buffer.  This double buffering process may seem bad, but
>> 		// has the advantage of allowing you to thread around data,
>> 		// process the buffer in chunks, etc.
>> 		data ~= buffer[0..num];
>> 	}
>>
>> 	socketStream.close;
>> 	socket.close;
>>
>> 	// It might be worthwhile to chunk this as well in some cases.
>> 	File file = new File("logo.gif", FileMode.Out);
>> 	file.write(data);
>> 	file.close;
>>
>> 	return 0;
>> }
> 
> Hey Travis,
> 
> I cannot begin to thank you enough for taking the time to explain this in such an exaustive example!  One reason I like the D community...I can ask rediculously simple questions and still people humor me with answers and take the time to care.  I was able to put together a working app based on your example.  The only part I don't understand from your example that strikes my curiousity is the buffer.  Why the 4096 size?  I guess what I don't understand is how do I know what buffer size I should use and if that is the best buffer size, why?  I hope that makes sense.
> 
> Thanks again,
> Zane

4k is the standard page size on x86 (well, on most machine actually), so 
I got in the habit of using it.  Normally I'd make sure it was page 
aligned as well (especially for output buffers), since under certain 
conditions and configurations FreeBSD can turn a socket write of a 4k 
page aligned buffer into a zero-copy operation.

Normally when you do a write(), the kernel will copy the data into it's 
own buffers for processing (so you can't mess with it before it is 
actually sent to the network).

FreeBSD (and possibly others) can do some page table trickery to mark 
the page ready only (well, copy on write), and when the time comes to 
send the data to the network card, if the page hasn't been modified, 
it'll send it straight from your (user space) buffer.

Anyway, it doesn't have to be 4k, or a multiple of 4k, in fact under 
certain configurations you may want it to be smaller (or larger).  In 
this case it doesn't really matter since using sockets stream is going 
to result in a mess of copies anyway (network card -> kernel buffer -> 
socket stream buffer -> your buffer).