Request for comments: std.d.lexer

[Jacob Carlborg]

On 2013-01-27 10:51, Brian Schott wrote:
> I'm writing a D lexer for possible inclusion in Phobos.
>
> DDOC:
> http://hackerpilot.github.com/experimental/std_lexer/phobos/lexer.html
> Code:
> https://github.com/Hackerpilot/Dscanner/blob/range-based-lexer/std/d/lexer.d
>
>
> It's currently able to correctly syntax highlight all of Phobos, but
> does a fairly bad job at rejecting or notifying users/callers about
> invalid input.
>
> I'd like to hear arguments on the various ways to handle errors in the
> lexer. In a compiler it would be useful to throw an exception on finding
> something like a string literal that doesn't stop before EOF, but a text
> editor or IDE would probably want to be a bit more lenient. Maybe having
> it run-time (or compile-time configurable) like std.csv would be the
> best option here.
>
> I'm interested in ideas on the API design and other high-level issues at
> the moment. I don't consider this ready for inclusion. (The current
> module being reviewed for inclusion in Phobos is the new std.uni.)

How about changing the type of TokenType to ushort, if all members fit. 
Just to minimize the size of a token.

-- 
/Jacob Carlborg