Implementing std.log

[Jonathan M Davis]

On 2011-05-08 15:45, Michel Fortin wrote:
> On 2011-05-08 18:24:12 -0400, Jonathan M Davis <jmdavisProg at gmx.com> said:
> >> On 5/8/2011 11:45 AM, Jose Armando Garcia wrote:
> >>> I am not sure I follow. Writing to disk is slower than writing to
> >>> memory so you want to hide some of the cost of logging by either
> >>> buffering the write/log requests or messaging the requests to another
> >>> thread to do the logging for you. Does that answer your question?
> >> 
> >> I thought that might be the reason.  Makes sense if you have so much
> >> logging that it's a significant bottleneck, but I can't believe people
> >> write code like that.
> > 
> > They do at least some of the time. Constructing strings to log can be
> > slow regardless of whether you're outputting them or not, and waiting
> > for them to be outputted can slow things down quite a bit. So, if you
> > add a lot of logging to your code to be sure of what's going on, it can
> > really slow things down, and sometimes you _need_ that sort of logging.
> > And sometimes, even if you don't or you can't afford it, someone naively
> > does it and causes a bottleneck.
> > 
> > A lot of it depends on what type of program you're writing and who's
> > writing it, but logging can be a definite bottleneck. And it can be very
> > surprising how much a bottleneck it can be. If you're smart about it,
> > you won't generally end up with logging being a large bottleneck, but
> > logging CPU-intensive code is generally problematic because it _will_
> > become a bottleneck, and when you have larger projects (especially with
> > a lot of people on them), it can become far too easy for logging code to
> > be called for more often than you'd expect.
> 
> True. But on the other hand, if your program crashes, you lost the most
> valuable log entries -- those just before the crash -- as they're
> waiting in the logger thread's queue when the crash happens. So I don't
> think logging in a separate thread should be the default.

Oh, I'm not claiming that using a separate thread is the best way to do 
things. In fact, my first inclination would be very much _not_ to use a 
separate thread for logging. I wasn't trying to claim that that was the best 
approach. I was just pointing out that logging can definitely become a major 
bottleneck.

Logging should be made as efficient as we reasonably can. What the best 
approach to that is, I don't know. Creating a separate thread might be it, or 
it might not. I doubt that it is, but I can see why someone would think of 
doing it. Your reason as to why not to do that though is a very good one.

- Jonathan M Davis