post on using go 1.5 and GC latency

[Laeeth Isharc via Digitalmars-d-learn]

https://medium.com/@robin.verlangen/billions-of-request-per-day-meet-go-1-5-362bfefa0911

We then started analyzing the behavior of our Go application. On 
average the application spent ~ 2ms per request, which was great! 
It gave us 98 milliseconds to spare for network overhead, SSL 
handshake, DNS lookups and everything else that makes the 
internet work.

Unfortunately the standard deviation of the latency was high, 
about 100 milliseconds. Meeting our SLA became a major gamble. 
With the “runtime” package of Go we started profiling the entire 
application and found out that garbage collection was the cause, 
resulting in 95-percentile latencies of 279 milliseconds…

We decided to rewrite big chunks of the application to generate 
minimal or no garbage at all. This effectively helped reduce the 
interval at which garbage collection froze the rest of 
application to do its cleanup magic. But we were still having 
issues, so we decided to add more nodes to stay within our SLA. 
With over 80K requests per second at peak times, even minimal 
garbage can become a serious issue.

...

Yesterday evening (19 August), the moment had finally arrived. A 
stable version 1.5 of Go was released, claiming:

The “stop the world” phase of the collector will almost always be 
under 10 milliseconds and usually much less.
Just a few hours after the release we rebuilt the application 
with the new version of Go 1.5 and ran our unit and functional 
tests; they all passed. It seemed too good to be true, so we put 
some effort in manually verifying the functionality. After a few 
hours we decided it was safe to release it to a single production 
node.