Request for comments: std.d.lexer

[deadalnix]

On Sunday, 27 January 2013 at 10:32:39 UTC, deadalnix wrote:
> Very happy to see that !
>
> Some remarks :
>  - Many parameters should be compile time parameters. Instead 
> of runtime.
>  - I'm not a big fan of byToken name, but let's see what others 
> think of it.
>  - I'm not sure this is the role of the lexer to process 
> __IDENTIFIER__ special stuffs.
>  - You need to provide a way to specify haw textual 
> representation of the token (ie value) is set. The best way to 
> do it IMO is an alias parameter that return a string when 
> called with a string (then the user can choose to keep the 
> value from original string, create a copy, always get the same 
> copy with the same string, etc . . .).
>  - Ideally, the location format should be configurable.
>  - You must return at least a forward range, and not an input 
> range, otherwize a lexer cannot lookahead.
>  - I'm not sure about making TokenRange a class.
>
> As a more positive note, I was just in need of something like 
> this today and wondered if such project was going to finally 
> happen. This is a great step forward ! Some stuff still need to 
> be polished, but who get everything right the first time ?

And the famous Job's « one last thing » : I'm not a big fan of 
having OPERATORS_BEGIN of the same type as regular token types. 
Now they make valid token. Why not provide a set of function like 
isOperator ?