[ENet-discuss] ENet-discuss Digest, Vol 126, Issue 1

[Blair Holloway]

I don't have exact numbers to hand, but the last game I did work on was
eight players, with around 100kbps uplink (12 kilobytes a second). Note
that this was a distributed, peer-to-peer simulation, so that cost was the
same for all the clients, as well.

Since you have an authoritative server, a larger target number makes sense.
I expect that your target of 50 kilobytes can be met for 8 players,
provided that your player updates are small (24-32 bytes per player per
packet).

- Blair


On Thu, Dec 5, 2013 at 10:24 AM, Jérémy Richert
<jeremy.richert1 at gmail.com>wrote:

> Thanks for your quick and detailed answers!
> I will follow your advices and lower the server cycle, it indeed seems to
> be a good way to have a better overall reactivity.
> As for compression, it may be interesting only for sending maps, but I
> will put this at the end of the network to-do list :)
>
> Just one additional question: do you know approximately the bandwidth
> usage on the server side for the games you have work on?
> My target is to be below 50 kB/s upload for 8 players and below 100 kB/s
> for 16 players, to be able to host such a server on a "classic" Internet
> line (in France at least, I do not know the max upload rate for other
> countries...)
>
> Best regards,
> Jeremy Richert
>
>
>
> On Wed, Dec 4, 2013 at 9:16 AM, <enet-discuss-request at cubik.org> wrote:
>
>> Send ENet-discuss mailing list submissions to
>>         enet-discuss at cubik.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://lists.cubik.org/mailman/listinfo/enet-discuss
>> or, via email, send a message with subject or body 'help' to
>>         enet-discuss-request at cubik.org
>>
>> You can reach the person managing the list at
>>         enet-discuss-owner at cubik.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of ENet-discuss digest..."
>>
>> Today's Topics:
>>
>>    1.  Bandwidth/latency optimization (J?r?my Richert)
>>    2. Re:  Bandwidth/latency optimization (Blair Holloway)
>>    3. Re:  Bandwidth/latency optimization (Lee Salzman)
>>
>>
>> ---------- Forwarded message ----------
>> From: "Jérémy Richert" <jeremy.richert1 at gmail.com>
>> To: enet-discuss at cubik.org
>> Cc:
>> Date: Tue, 3 Dec 2013 22:45:49 +0100
>> Subject: [ENet-discuss] Bandwidth/latency optimization
>> Hi all,
>>
>> I am currently developing a multiplayer FPS, and I am wondering how to
>> tune ENet to have a network engine as optimized as possible on a bandwidth
>> usage and a latency point of view.
>> I would like to have your thougts on the following aspects:
>>
>> ----------------
>> 1. Packet size
>> ----------------
>>
>> I was wondering whether it was better to send large packets or to divide
>> them into small packets.
>>
>> Pros of the large packets :
>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for IPv4,
>> 10 for ENet)
>>
>> Pros of the small packets :
>> - When a packet is lost, less data is lost
>> - When a reliable packet is lost, resending it requires less bandwidth
>> - Lower latency
>>
>> From what I have read, most people agree that it is recommended to send
>> small packets to avoid packet splitting. This means that the application
>> has to ensure that the size of the data sent does not exceed the MTU. Also,
>> as the MTU depends on the router, some people recommended to have packets
>> that will never be splitted, i.e. <= 576 bytes.
>> What is your experience on this point?
>> For now I have capped the data size to 1500, but I am thinking of
>> reducing it for a better latency. Does anyone know the typical packet loss
>> rate?
>>
>> ----------------
>> 2. Compression
>> ----------------
>>
>> Has anyone used compression in a network engine? If yes, which
>> compression algorithm? What was the average gain?
>> I have read that John Carmack used the Huffman compression in the Quake 3
>> network engine because it was well suited for network data compression, but
>> I still need to find some time to implement it in my program and do some
>> tests.
>>
>> ----------------
>> 3. Channels
>> ----------------
>>
>> What is your network channel policy? How many channels do you use? What
>> do you send on each channel?
>> At the moment I have 2 channels: one for sending unreliable data (a lot),
>> another one for reliable data.
>> I have chosen this organization to avoid blocking the unreliable data
>> while waiting for an ACK for the reliable data. I am thinking of adding
>> another channel to send high priority reliable data, but I am not sure of
>> the benefit as I already group the reliable data before sending them to
>> limit the blocking. It may be useful if the packet loss rate is too high.
>>
>> ----------------
>> 4. Server cycle
>> ----------------
>>
>> I know this aspect depends on the game type, but I would be interested in
>> knowing how your applications work on this.
>> On my side, based on Valve's introduction to network concepts (+ some
>> readings on the UT and Quake network engines), I have decided to implement
>> a 50-ms cycle on the server side. This means that the servers only updates
>> the simulation and notifies the clients each 50 ms. Meanwhile, it only
>> reads the network messages to empty the network event buffer and to handle
>> the reliable packet sending.
>> On the client side, I have introduced a voluntary delay of 70 ms, which
>> goes unnoticed on a user point of view, but helps a lot for overall
>> fluidity as the client will almost always have the following world update
>> (unless a packet is lost).
>> I will soon try to increase the server cycle time to 60 ms to gain 10-20%
>> of bandwidth. I also plan to separate the display delay of the player (it
>> will stay to 70 ms) and the rest of the world (increased to 100 ms).
>> However, I am afraid of the impact on reactivity.
>>
>> Does anyone have a similar architecture? What is your experience on
>> timings?
>> If not, has anyone already developed a FPS or a game very dependant on
>> the network speed? If yes, what are your advices?
>>
>> ----------------
>> 5. Other
>> ----------------
>>
>> If anyone has some useful advices on how to improve the network engine of
>> a game/application, I would be pleased to hear it (well, to read it at
>> least).
>>
>>
>> Thanks in advance for all your experience sharing.
>>
>> Best regards,
>> Jeremy Richert
>>
>>
>> ---------- Forwarded message ----------
>> From: Blair Holloway <thynameisblair at chaosandcode.com>
>> To: Discussion of the ENet library <enet-discuss at cubik.org>
>> Cc:
>> Date: Tue, 3 Dec 2013 15:16:54 -0800
>> Subject: Re: [ENet-discuss] Bandwidth/latency optimization
>> Here are my thoughts:
>>
>> 1) Packet size
>>
>> Smaller is nearly universally better, especially if any of your data are
>> reliable, as your packets may quickly be eaten up by resent data over poor
>> connections.
>>
>> A second advantage of smaller data sizes is that you can generally have a
>> higher number of maximum players, or more stuff happening in your game.
>>
>> 2) Compression
>>
>>  Of all the games I've worked on, we only ever used bit-packing or a
>> range encoder to compress the data; we never used a more general-purpose
>> compression algorithm.
>>
>> One exception was a game whose packets often contained a lot of repeated
>> string data. We generated a dictionary at runtime using previously seen
>> strings, and used indices into that dictionary during future transmissions.
>>
>> In general, the best "compression" we implemented was always "remove the
>> need for X to be sent". :)
>>
>> 3) Channels
>>
>> Can you envision a need up-front for high-priority reliable data? I've
>> never encountered such a case myself, but I *have* found cases where
>> certain unreliable data need to be prioritized ahead of others. Sometimes
>> this can be implemented as simply as "put the high priority stuff in the
>> packet first; continue until pre-set limit reached".
>>
>> 4) Server cycle
>>
>> The only thing I can really say here is that the smaller simulation time
>> step your server takes, the lesser your latency will be. Increasing the
>> timestep will save you bandwidth, but it can make your game seem laggier.
>> For an FPS, you probably want to aim for an even higher packet rate -- 30
>> packets per second wouldn't be unusual.
>>
>> Cheers,
>>
>> - Blair
>>
>>
>>
>> On Tue, Dec 3, 2013 at 1:45 PM, Jérémy Richert <jeremy.richert1 at gmail.com
>> > wrote:
>>
>>> Hi all,
>>>
>>> I am currently developing a multiplayer FPS, and I am wondering how to
>>> tune ENet to have a network engine as optimized as possible on a bandwidth
>>> usage and a latency point of view.
>>> I would like to have your thougts on the following aspects:
>>>
>>> ----------------
>>> 1. Packet size
>>> ----------------
>>>
>>> I was wondering whether it was better to send large packets or to divide
>>> them into small packets.
>>>
>>> Pros of the large packets :
>>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for
>>> IPv4, 10 for ENet)
>>>
>>> Pros of the small packets :
>>> - When a packet is lost, less data is lost
>>> - When a reliable packet is lost, resending it requires less bandwidth
>>> - Lower latency
>>>
>>> From what I have read, most people agree that it is recommended to send
>>> small packets to avoid packet splitting. This means that the application
>>> has to ensure that the size of the data sent does not exceed the MTU. Also,
>>> as the MTU depends on the router, some people recommended to have packets
>>> that will never be splitted, i.e. <= 576 bytes.
>>> What is your experience on this point?
>>> For now I have capped the data size to 1500, but I am thinking of
>>> reducing it for a better latency. Does anyone know the typical packet loss
>>> rate?
>>>
>>> ----------------
>>> 2. Compression
>>> ----------------
>>>
>>> Has anyone used compression in a network engine? If yes, which
>>> compression algorithm? What was the average gain?
>>> I have read that John Carmack used the Huffman compression in the Quake
>>> 3 network engine because it was well suited for network data compression,
>>> but I still need to find some time to implement it in my program and do
>>> some tests.
>>>
>>> ----------------
>>> 3. Channels
>>> ----------------
>>>
>>> What is your network channel policy? How many channels do you use? What
>>> do you send on each channel?
>>> At the moment I have 2 channels: one for sending unreliable data (a
>>> lot), another one for reliable data.
>>> I have chosen this organization to avoid blocking the unreliable data
>>> while waiting for an ACK for the reliable data. I am thinking of adding
>>> another channel to send high priority reliable data, but I am not sure of
>>> the benefit as I already group the reliable data before sending them to
>>> limit the blocking. It may be useful if the packet loss rate is too high.
>>>
>>> ----------------
>>> 4. Server cycle
>>> ----------------
>>>
>>> I know this aspect depends on the game type, but I would be interested
>>> in knowing how your applications work on this.
>>> On my side, based on Valve's introduction to network concepts (+ some
>>> readings on the UT and Quake network engines), I have decided to implement
>>> a 50-ms cycle on the server side. This means that the servers only updates
>>> the simulation and notifies the clients each 50 ms. Meanwhile, it only
>>> reads the network messages to empty the network event buffer and to handle
>>> the reliable packet sending.
>>> On the client side, I have introduced a voluntary delay of 70 ms, which
>>> goes unnoticed on a user point of view, but helps a lot for overall
>>> fluidity as the client will almost always have the following world update
>>> (unless a packet is lost).
>>> I will soon try to increase the server cycle time to 60 ms to gain
>>> 10-20% of bandwidth. I also plan to separate the display delay of the
>>> player (it will stay to 70 ms) and the rest of the world (increased to 100
>>> ms). However, I am afraid of the impact on reactivity.
>>>
>>> Does anyone have a similar architecture? What is your experience on
>>> timings?
>>> If not, has anyone already developed a FPS or a game very dependant on
>>> the network speed? If yes, what are your advices?
>>>
>>> ----------------
>>> 5. Other
>>> ----------------
>>>
>>> If anyone has some useful advices on how to improve the network engine
>>> of a game/application, I would be pleased to hear it (well, to read it at
>>> least).
>>>
>>>
>>> Thanks in advance for all your experience sharing.
>>>
>>> Best regards,
>>> Jeremy Richert
>>>
>>> _______________________________________________
>>> ENet-discuss mailing list
>>> ENet-discuss at cubik.org
>>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>>
>>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Lee Salzman <lsalzman at gmail.com>
>> To: Discussion of the ENet library <enet-discuss at cubik.org>
>> Cc:
>> Date: Wed, 4 Dec 2013 10:16:40 +0200
>> Subject: Re: [ENet-discuss] Bandwidth/latency optimization
>> My experiences as for Cube/Sauerbraten:
>>
>> 1. packet size
>>
>> Do not hand ENet lots of tiny packets. This is bad. Use the ENet packets
>> as batches, i.e. take a bunch of your little user-level with their own
>> specialized little headers (in Sauerbraten these headers are only 1 byte
>> per packet!). However, keep these batch sizes underneath the ENet MTU with
>> some small reserved space for ENet headers, i.e. peer->mtu - 100 should be
>> good for limiting.
>>
>> The reason this became important for Sauerbraten is we were sending
>> *lots* of these, so ENet packet headers would have become a significant
>> fraction of the total size, since the payloads of these packets are small.
>> That Sauerbraten's internal protocol can get this down to 1 byte sub-header
>> essentially gets rid of that issue. These are then batched into
>> peer->mtu-100 ENet packets as described above to avoid triggering ENet's
>> transparent fragmentation system.
>>
>> 2. compression
>>
>> Pack your data intelligently in the first place, and you won't get much
>> if any benefit from compression. At least Sauerbraten's packets get at most
>> only 5-10% more out of ENet's range coder because we encode, say, integers
>> using an extension scheme that uses as few bytes as necessary (not unlike
>> how, say, UTF8 works). So, just be as smart as possible on the encoding up
>> front, and you can mostly avoid compression and save a lot of the CPU time
>> that you would have normally spent on that.
>>
>> Failing that, others have been happy enough to stick with ENet's built-in
>> range coder.
>>
>> 3. channels
>>
>> Nothing wrong with a 2 channel setup, and is almost what I use for
>> Sauerbraten. I send bulk movement data on a channel that mostly is only
>> unreliable. Occasionally reliable stuff gets sequenced into that, but very
>> rarely (only for special events like teleports or jumppads). The other
>> channel I use for reliable stuff like gun shots, chat, etc. There are other
>> schemes like how Quake 3 works that can be used to send even that
>> 'reliable' data as unreliable, but it wasn't terribly worth it for
>> Sauerbraten's case.
>>
>> I also have another channel or two I use for file transfers and things
>> that I don't want stalling the flow of everything else for long durations.
>>
>> 4. server-cycle
>>
>> Sauerbraten just because of its insane movement speed uses 33ms (~30 Hz).
>> Whereas for variants with slightly saner move speed, like Red
>> Eclipse/Tesseract/AssaultCube, I have preferred to settle on 40ms (25 Hz).
>> What matters most for this is the quality of your physics interpolation and
>> also that you do a lot of client-side prediction rather than waiting on
>> round-trips, then these small changes in rate don't matter quite as much,
>> but higher fidelity of input to the interpolation always helps.
>>
>>
>>
>>
>> On Tue, Dec 3, 2013 at 11:45 PM, Jérémy Richert <
>> jeremy.richert1 at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am currently developing a multiplayer FPS, and I am wondering how to
>>> tune ENet to have a network engine as optimized as possible on a bandwidth
>>> usage and a latency point of view.
>>> I would like to have your thougts on the following aspects:
>>>
>>> ----------------
>>> 1. Packet size
>>> ----------------
>>>
>>> I was wondering whether it was better to send large packets or to divide
>>> them into small packets.
>>>
>>> Pros of the large packets :
>>> - Less overhead due to the protocols (8 bytes for UDP, 20 bytes for
>>> IPv4, 10 for ENet)
>>>
>>> Pros of the small packets :
>>> - When a packet is lost, less data is lost
>>> - When a reliable packet is lost, resending it requires less bandwidth
>>> - Lower latency
>>>
>>> From what I have read, most people agree that it is recommended to send
>>> small packets to avoid packet splitting. This means that the application
>>> has to ensure that the size of the data sent does not exceed the MTU. Also,
>>> as the MTU depends on the router, some people recommended to have packets
>>> that will never be splitted, i.e. <= 576 bytes.
>>> What is your experience on this point?
>>> For now I have capped the data size to 1500, but I am thinking of
>>> reducing it for a better latency. Does anyone know the typical packet loss
>>> rate?
>>>
>>> ----------------
>>> 2. Compression
>>> ----------------
>>>
>>> Has anyone used compression in a network engine? If yes, which
>>> compression algorithm? What was the average gain?
>>> I have read that John Carmack used the Huffman compression in the Quake
>>> 3 network engine because it was well suited for network data compression,
>>> but I still need to find some time to implement it in my program and do
>>> some tests.
>>>
>>> ----------------
>>> 3. Channels
>>> ----------------
>>>
>>> What is your network channel policy? How many channels do you use? What
>>> do you send on each channel?
>>> At the moment I have 2 channels: one for sending unreliable data (a
>>> lot), another one for reliable data.
>>> I have chosen this organization to avoid blocking the unreliable data
>>> while waiting for an ACK for the reliable data. I am thinking of adding
>>> another channel to send high priority reliable data, but I am not sure of
>>> the benefit as I already group the reliable data before sending them to
>>> limit the blocking. It may be useful if the packet loss rate is too high.
>>>
>>> ----------------
>>> 4. Server cycle
>>> ----------------
>>>
>>> I know this aspect depends on the game type, but I would be interested
>>> in knowing how your applications work on this.
>>> On my side, based on Valve's introduction to network concepts (+ some
>>> readings on the UT and Quake network engines), I have decided to implement
>>> a 50-ms cycle on the server side. This means that the servers only updates
>>> the simulation and notifies the clients each 50 ms. Meanwhile, it only
>>> reads the network messages to empty the network event buffer and to handle
>>> the reliable packet sending.
>>> On the client side, I have introduced a voluntary delay of 70 ms, which
>>> goes unnoticed on a user point of view, but helps a lot for overall
>>> fluidity as the client will almost always have the following world update
>>> (unless a packet is lost).
>>> I will soon try to increase the server cycle time to 60 ms to gain
>>> 10-20% of bandwidth. I also plan to separate the display delay of the
>>> player (it will stay to 70 ms) and the rest of the world (increased to 100
>>> ms). However, I am afraid of the impact on reactivity.
>>>
>>> Does anyone have a similar architecture? What is your experience on
>>> timings?
>>> If not, has anyone already developed a FPS or a game very dependant on
>>> the network speed? If yes, what are your advices?
>>>
>>> ----------------
>>> 5. Other
>>> ----------------
>>>
>>> If anyone has some useful advices on how to improve the network engine
>>> of a game/application, I would be pleased to hear it (well, to read it at
>>> least).
>>>
>>>
>>> Thanks in advance for all your experience sharing.
>>>
>>> Best regards,
>>> Jeremy Richert
>>>
>>> _______________________________________________
>>> ENet-discuss mailing list
>>> ENet-discuss at cubik.org
>>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>>
>>>
>>
>> _______________________________________________
>> ENet-discuss mailing list
>> ENet-discuss at cubik.org
>> http://lists.cubik.org/mailman/listinfo/enet-discuss
>>
>>
>
> _______________________________________________
> ENet-discuss mailing list
> ENet-discuss at cubik.org
> http://lists.cubik.org/mailman/listinfo/enet-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cubik.org/pipermail/enet-discuss/attachments/20131205/a0c7e258/attachment-0001.html>