[ENet-discuss] Peer starving problem....

[Nono BEZMan]

Thanx Lee for chiming in.

Two things seem to have greatly improved on my side:

- I set the Max window size to 4KB instead of the
orginial 32K. I think this is what was causing most of
the starvation.

- In enet_protocol_check_timeouts() now uses '+=
peer->roundTripTimeVariance;' instead of '*=2;' as
follows (to give more time to the peers to adjust):

//outgoingCommand -> roundTripTimeout *= 2;
outgoingCommand -> roundTripTimeout +=
peer->roundTripTimeVariance;

- Also, I had already changed the SO_SNDBUF socket
option to be the same as the receive buffer size.

Now, I am able to stream packets of 512KB with 20
peers simultaneously, whereas 3 peers would show
starvation and disconnect two after a few seconds. [I
am using one client on a Business Yahoo DSL, and one
server on our T1].

I still have two small issues I am investigating. For
some reasons, sometimes a peer seem to be stuck on a
command and send it indifinitely. It appears as it
still processes other messages, but it will continue
sending this command nevertheless (happens when you
connect 200 peers to the server. All with be connected
on both sides, but some peers on the client will still
send CONNECT cmds in a loop!?!).

Also, it appears that when I was testing 250
connections (no packet transfered - idle connections),
from one client to a server, all the pings where
occuring at the same time, even though the connections
were progressively made over a period a 20 seconds or
so. Note: I had to change the PING_INTERVAL to 5000
btw.

Anyways, thanx again Lee for this great library, and i
hope my findings may help some people.


--- Lee Salzman <lsalzman1 at cox.net> wrote:

> There is an in-flight window for reliable data,
> meaning that for any 
> given peer, only some fixed amount of data can be
> going over the network 
> at any given time. So even if you stuff 512KB into
> one peer, it won't 
> send it all out in one call to
> enet_protocol_send_outgoing_commands().
> 
> However, this window is per-peer, so if you have 50
> peers, you will 
> multiply the amount of data sent in 
> enet_protocol_send_outgoing_commands() and likewise
> the amount of time 
> that function takes to execute. While it is stalled
> in there sending it 
> can't acknowledge any connections, hence why peers
> are timing out.
> 
> There are a few feasible ways around this. The first
> is to just try 
> making the send buffer in the socket a lot larger by
> setting the 
> SO_SNDBUF socket option (something I should probably
> be doing anyway). 
> However, if you just keep pumping data at an
> overwhelming rate that 
> would still probably not do much.
> 
> The other way is to limit the total amount of data 
> enet_protocol_send_outgoing_commands() is allowed to
> send out in one 
> call, so that even if it overflows the send buffer
> it won't wait very 
> long at all.
> 
> The whole throttling mechanism currently in place
> for reliable data, as 
> I have said before, is kinda screwy and really does
> need to be rewritten 
> in the future, but I'm still working on that.
> 
> Lee
> 
> Nono BEZMan wrote:
> > This is what I feared... enet does cope well with
> > over-the-bandwidth tranfers. I thought it would do
> > some flow control with the ack/retry system, but
> it
> > does not seem to do it fairly.
> > 
> > I also tried to put a bandwidth limit in
> > enet_host_create, but it did not do anything. The
> > throtle_bandwidth() function is called, but it
> does
> > not seem to slow down the whole connection.
> > 
> > Again, what I do not get is why sending 512KB on
> ONE
> > peer (which means having to resend out-of-order
> > packets) works fine where sending 1KB (<MTU) on 50
> > peers does not. The bandwidth used is exactly the
> > same!
> > 
> > Packet loss shold be the same in both cases
> (except if
> > something weird is going on with the rtt on some
> > peers, which does not happen when you only have
> one
> > peer). The 50-peer case should even be easier on
> > enet!!!
> > 
> > I am really looking for solutions here, as I do
> think
> > it is a tweak to enet that would really be worth
> it.
> > So much work has already been done in it, and we
> are
> > close to a perfect solution here... except for
> that
> > starvation issue.
> > 
> > Thanx again.
> > 
> _______________________________________________
> ENet-discuss mailing list
> ENet-discuss at cubik.org
> http://lists.cubik.org/mailman/listinfo/enet-discuss
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com