Request for comments: std.d.lexer

[Dejan Lekic]

On Sunday, 27 January 2013 at 09:51:10 UTC, Brian Schott wrote:
> I'm writing a D lexer for possible inclusion in Phobos.
>
> DDOC: 
> http://hackerpilot.github.com/experimental/std_lexer/phobos/lexer.html
> Code: 
> https://github.com/Hackerpilot/Dscanner/blob/range-based-lexer/std/d/lexer.d
>
> It's currently able to correctly syntax highlight all of 
> Phobos, but does a fairly bad job at rejecting or notifying 
> users/callers about invalid input.
>
> I'd like to hear arguments on the various ways to handle errors 
> in the lexer. In a compiler it would be useful to throw an 
> exception on finding something like a string literal that 
> doesn't stop before EOF, but a text editor or IDE would 
> probably want to be a bit more lenient. Maybe having it 
> run-time (or compile-time configurable) like std.csv would be 
> the best option here.
>
> I'm interested in ideas on the API design and other high-level 
> issues at the moment. I don't consider this ready for 
> inclusion. (The current module being reviewed for inclusion in 
> Phobos is the new std.uni.)

Having a "standard" lexer module, that many people work on, and 
has up-to-date rules, is priceless! Thank you for doing this.