std.d.lexer: pre-voting review / discussion

[Martin Nowak]

On 09/11/2013 05:01 PM, Dicebot wrote:
> std.d.lexer is standard module for lexing D code, written by Brian Schott
>
Nice!

> Please take this part seriously: "If you identify problems along the
> way, please note if they are minor, serious, or showstoppers."
> (http://wiki.dlang.org/Review/Process). This information later
> will be used to determine if library is ready for voting.
>
I only had a short look at the source code and was left with a good 
impression, i.e. it looks like a lexer written in D.

I do like how you made the lexer configurable to serve the different 
needs. Also it was very easy to setup and use.

The appended underscore for keywords is much nicer than messing with 
uppercase/lowercase which was what I did.

One thing that bothered me was the usage of const(ubyte)[] as input.
If you operate on UTF-8 it should take const(char)[] you can cast it to 
const(ubyte)[] internally.

Also a convenience function that reads a file and processes UTF BOM 
marks would be nice (see toUtf8 
https://github.com/dawgfoto/lexer/blob/master/dlexer/dlexer.d#L1429), 
but that could as well fit somewhere else into phobos.

> ---- Information request from module author ----
>
> Performance was a major discussed topic in previous thread. Could you
> please provide benchmarking data for version currently ongoing the review?

Nice too, it works as fast as my D lexer 
(https://github.com/dawgfoto/lexer) which is about as fast as DMD's 
lexer (when compiled with LDC).