std.d.lexer requirements

[Jonathan M Davis]

On Thursday, August 02, 2012 11:06:39 Walter Bright wrote:
> On 8/2/2012 4:47 AM, deadalnix wrote:
> > Le 02/08/2012 10:13, Walter Bright a écrit :
> >> As fast as the dmd one would be best.
> > 
> > That'd be great but . . .
> > 
> > lexer really isn't the performance bottleneck of dmd (or any compiler of a
> > non trivial language). Additionally, anybody that have touched dmd source
> > code can agree that usability/maintainability isn't as its best.
> > 
> > Sacrificing some perfs in a non bottleneck area to increase ease of use
> > make perfect sense.
> 
> A lexer is like a device driver, or a memory allocator. Having it be as fast
> as possible pays off dividends endlessly for whoever uses it.
> 
> And in my profiling, lexing speed is a big factor in the debug build times.
> 
> The Digital Mars C++ compiler (and its antecedents) made a lot of hay by
> being the fastest compiler available by a wide margin. A big advantage D
> has is also compile speed.
> 
> Please do not underestimate its importance.

Recent benchmarks by Iain Buclaw show that lexing is not a bottleneck in dmd, 
so it's natural for some folks to think that the lexing is not all that 
critical in comparison to other compiler components. However, the reason that 
lexing is not a bottleneck is almost ceratinly because the lexer is so 
thoroughly optimized. So, if it _didn't_ try and be as efficient as possible, it 
_would_ be a bottleneck.

However, given that we're providing an API for a lexer rather than having the 
lexer be an internal component of a compiler, that _does_ put more pressure on 
making the API user-friendly. Hopefully that can be done without sacrificing 
any performance, but there may be a point where the only way to make it faster 
would be to make the API very unfriendly. If that's the case, then we're going 
to have some tough decisions to make. But we'll cross that bridge if/when we 
get to it.

- Jonathan M Davis