Formal review of std.lexer

[Dicebot]

http://wiki.dlang.org/Review/std.lexer

This is follow-up by Brian to his earlier proposal 
(http://wiki.dlang.org/Review/std.d.lexer). This time proposed 
module focuses instead on generic lexer generation as discussed 
in matching voting thread.

Docs: 
http://hackerpilot.github.io/experimental/std_lexer/phobos/lexer.html
Code: 
https://github.com/Hackerpilot/Dscanner/blob/master/stdx/lexer.d

Example topics to evaluate during review:
  - Feasibility of overall design and concept
  - Quality of documentation
  - Consistency with existing Phobos modules
  - Overall quality of implementation

Initial review will end on March the 8th.