Request for comments: std.d.lexer

[deadalnix]

Very happy to see that !

Some remarks :
  - Many parameters should be compile time parameters. Instead of 
runtime.
  - I'm not a big fan of byToken name, but let's see what others 
think of it.
  - I'm not sure this is the role of the lexer to process 
__IDENTIFIER__ special stuffs.
  - You need to provide a way to specify haw textual 
representation of the token (ie value) is set. The best way to do 
it IMO is an alias parameter that return a string when called 
with a string (then the user can choose to keep the value from 
original string, create a copy, always get the same copy with the 
same string, etc . . .).
  - Ideally, the location format should be configurable.
  - You must return at least a forward range, and not an input 
range, otherwize a lexer cannot lookahead.
  - I'm not sure about making TokenRange a class.

As a more positive note, I was just in need of something like 
this today and wondered if such project was going to finally 
happen. This is a great step forward ! Some stuff still need to 
be polished, but who get everything right the first time ?