Network server design question

[Sean Kelly]

On Aug 4, 2013, at 12:38 PM, Marek Janukowicz <marek at janukowicz.net> wrote:

> I'm writing a network server with some specific requirements:
> - 5-50 clients connected (almost) permanently (maybe a bit more, but 
> definitely not hundreds of them)
> - possibly thousands of requests per seconds
> - responses need to be returned within 5 seconds or the client will 
> disconnect and complain

Given the relatively small number of concurrent connections, you may be best off just spawning a thread per connection.  The cost of context switching at that level of concurrency is reasonably low, and the code will be a heck of a lot simpler than an event loop dispatching jobs to a thread pool (which is the direction you might head with a larger number of connections).


> Currently I have a Master thread (which is basically the main thread) which 
> is handling connections/disconnections, socket operations, sends parsed 
> requests for processing to single Worker thread, sends responses to clients. 
> Interaction with Worker is done via message passing.
> 
> The problem with my approach is that I read as much data as possible from 
> each ready client in order. As there are many requests this read phase might 
> take a few seconds making the clients disconnect.

This seems weird to me.  Are those reads blocking for some length of time?  I would expect them to return pretty much instantly.  How much data is in each request?


> Now I see 2 possible solutions:
> 
> 1. Stay with the design I have, but change the workflow somewhat - instead 
> of reading all the data from clients just read some requests and then send 
> responses that are ready and repeat; the downside is that it's more 
> complicated than current design, might be slower (more loop iterations with 
> less work done in each iteration) and might require quite a lot of tweaking 
> when it comes to how many requests/responses handle each time etc.

There are a bunch of different approaches along these lines, but the crux of it is that you'll be multiplexing N connections across an M-sized thread pool.  Each connection carries a buffer with it, and whenever data is available you stick that connection in a work queue, and let a pooled thread accumulate the new data into that connection's buffer and potentially process the complete request.


> 2. Create separate thread per each client connection. I think this could 
> result in a nice, clean setup, but I see some problems:
> - I'm not sure how ~50 threads will do resource-wise (although they will 
> probably be mostly waiting on Socket.select)

With a thread per connection you can probably just do blocking reads in each thread and not bother with select at all.  And with only 50 threads I don't think you'll see a performance problem.  I've been reading up on Java NIO recently (their approach for supporting epoll within Java), and some people have actually said that the old thread-per-connection approach was actually faster in their tests.  Of course, no one seems to test beyond a few thousand concurrent connections, but that's still well above what you're doing.  In short, I'd consider benchmarking it and see if performance is up to snuff.


> - I can't initialize threads created via std.concurrency.spawn with a Socket 
> object ("Aliases to mutable thread-local data not allowed.")

You can cast the Socket to shared and cast away shared upon receipt.  I'd like a more formal means of moving uniquely referenced data via std.concurrency, but that will do the trick for now.


> - I already have problems with "interrupted system call" on Socket.select 
> due to GC kicking in; I'm restarting the call manually, but TBH it sucks I 
> have to do anything about that and would suck even more to do that with 50 
> or so threads

Just wrap it in a function that tests the return value and loops if necessary.  Plenty of system calls need to deal with the EINTR error.  It may not just be GC that's causing it.  There's a decent chance you'll have to deal with SIGPIPE as well.