[ENet-discuss] Death by a thousand cuts

[Lee Salzman]

It is astounding sometimes what you figure out by doing some simple 
number crunching.

In Sauerbraten, the client updates the server with its world position 30 
times a second. The server then broadcasts this position to all other 
clients. Thus each client is receiving 30*(N-1) position packets per 
second, and the server is sending out 30*(N-1)*N position packets per 
second to client. Now, I've known this forever, but it only occurred 
recently that players started noticing worse performance. Then it dawned 
on me that we had made position packets larger by 6-10 bytes recently.

So if we have 6 clients, for example, that's only 900 packets per second 
being sent out. Seems harmless, right? Well, that's until I realized the 
size of our packets plus the protocol overhead (not counting the UDP 
headers) was coming out to well over 50 bytes. So in that situation the 
server is sending out a hefty 45 KB/s and the client is receiving a nice 
7.5 KB/s. Every single byte of packet size is multiplied by 900, and it 
adds up quick.

The moral of that story is: lots of packets * small number of bytes = 
tons of wasted bandwidth.

So I went through ENet and managed to shave off a minimum of 12 bytes 
from the protocol headers. In the context of the above example, that 
saves the client from sending an extra 1.8 KB/s and the server from 
sending 10.8 KB/s, minimum. It required some trickery to fit 32 bit time 
stamps and sequence numbers into 16 bits, but it seemed to work out 
rather well.

In the process of this, I also cut out certain fields of the header that 
were redundant. But to prevent packets from being more prone to 
corruption from that, I added in a cheap CRC32 checksum (enabled by 
configure option), which should actually make packets less so. I know 
that will make some users happy.

I also realized that under that much load the packet throttle really 
doesn't handle as smoothly as I would like. So I may be rewriting the 
throttle to work in the manner I wanted it to ages ago.

So, FYI: there may be a 1.1 coming out soon.

P.S. I promise I will start keeping change logs as of 1.0. :)