Network server design question

[Regan Heath]

On Sun, 04 Aug 2013 20:38:49 +0100, Marek Janukowicz  
<marek at janukowicz.net> wrote:
> I'm writing a network server with some specific requirements:
> - 5-50 clients connected (almost) permanently (maybe a bit more, but
> definitely not hundreds of them)
> - possibly thousands of requests per seconds
> - responses need to be returned within 5 seconds or the client will
> disconnect and complain
>
> Currently I have a Master thread (which is basically the main thread)  
> which
> is handling connections/disconnections, socket operations, sends parsed
> requests for processing to single Worker thread, sends responses to  
> clients.
> Interaction with Worker is done via message passing.
>
> The problem with my approach is that I read as much data as possible from
> each ready client in order. As there are many requests this read phase  
> might
> take a few seconds making the clients disconnect. Now I see 2 possible
> solutions:
>
> 1. Stay with the design I have, but change the workflow somewhat -  
> instead
> of reading all the data from clients just read some requests and then  
> send
> responses that are ready and repeat; the downside is that it's more
> complicated than current design, might be slower (more loop iterations  
> with
> less work done in each iteration) and might require quite a lot of  
> tweaking
> when it comes to how many requests/responses handle each time etc.
>
> 2. Create separate thread per each client connection. I think this could
> result in a nice, clean setup, but I see some problems:
> - I'm not sure how ~50 threads will do resource-wise (although they will
> probably be mostly waiting on Socket.select)
> - I can't initialize threads created via std.concurrency.spawn with a  
> Socket
> object ("Aliases to mutable thread-local data not allowed.")
> - I already have problems with "interrupted system call" on Socket.select
> due to GC kicking in; I'm restarting the call manually, but TBH it sucks  
> I
> have to do anything about that and would suck even more to do that with  
> 50
> or so threads
>
> If anyone has any idea how to handle the problems I mentioned or has any
> idea for more suitable design I would be happy to hear it. It's also
> possible I'm approaching the issue from completely wrong direction, so  
> you
> can correct me on that as well.

Option #2 should be fine, provided you don't intend to scale to a larger  
number of clients.  I have had loads of experience with server  
applications on Windows and a little less on the various flavours of  
UNIXen and 50 connected clients serviced by 50 threads should be perfectly  
manageable for the OS.

It sounds like only non-blocking sockets have the GC interrupt issue, if  
so use non-blocking sockets instead.  However, it occurs to me that the  
issue may rear it's head again on the call to select() on non-blocking  
sockets, so it is worth testing this first.

If there is no way around the GC interrupt issue then code up your own  
recv function and re-use it all your threads, not ideal but definitely  
workable.

In the case of non-blocking sockets your read operation needs to account  
for the /this would block/ error code, and should go something like this..  
(using low level socket function call names because I have not used the D  
socket library recently)
1. Attempt recv(), expect either DATA or ERROR.
1a. If DATA, process data and handle possible partial request(s) - by  
buffering and going back to #1 for example
1b. If ERROR, check for code [WSA]EWOULDBLOCK and if so go to step #2
1c. If ERROR and not would block, fail/exit/disconnect.
2. Perform select() (**this may be interruptable by GC**) for a finite  
shortish timeout - if you want your client handlers to react quickly to  
the signal to shutdown then you want a shorter time - for example.
2a. If select returns a read indicator, go to #1
2b. If select returns an error, fail/exit/disconnect.
2c. Otherwise, go to #2

Do you have control of the connecting client code as well?  If so, think  
about disabling the Nagle algorithm:
http://en.wikipedia.org/wiki/Nagle's_algorithm

You will want to ensure the client writes it's requests in a single send()  
call but in this way you reduce the delay in receiving requests at the  
server side.  If the client writes multiple requests rapidly then with  
Nagle enabled it may buffer them on the client end and will delay the  
server seeing the first, but with it disabled the server will see the  
first as soon as it is written and can start processing it while the  
client writes.  So depending on how your clients send requests, you may  
see a performance improvement here.

I don't know how best to solve the "Aliases to mutable thread-local data  
not allowed.".  You will need to ensure the socket is allocated globally  
(not thread local) and because you know it's unique and not shared you can  
cast it as such to get it into the thread, once there you can cast it back  
to unshared/local/mutable.  Not ideal, but not illegal or invalid AFAICS.

FYI.. For a better more scaleable solution you would use async IO with a  
pool of worker threads, I am not sure if D has good support for this and  
it's been a while since I did this myself (outside of using the nice C#  
library support for it).

Regan

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/