std.d.lexer: pre-voting review / discussion

[Dicebot]

std.d.lexer is standard module for lexing D code, written by 
Brian Schott

---- Input ----

Code: https://github.com/Hackerpilot/phobos/tree/master/std/d

Documentation:
http://hackerpilot.github.io/experimental/std_lexer/phobos/lexer.html

Initial discussion:
http://forum.dlang.org/thread/dpdgcycrgfspcxenzrjf@forum.dlang.org

Usage example in real project:
https://github.com/Hackerpilot/Dscanner
(as stdx.d.lexer, Brian, please correct me if versions do not 
match)

---- Information for reviewers ----

(yes, I am mostly copy-pasting this :P)

Goal of this thread is to detect if there are any outstanding
issues that need to fixed before formal "yes"/"no" voting
happens. If no critical objections will arise, voting will begin
starting with a next week. Otherwise it depends on time module 
author needs to implement suggestions.

Please take this part seriously: "If you identify problems along 
the
way, please note if they are minor, serious, or showstoppers."
(http://wiki.dlang.org/Review/Process). This information later
will be used to determine if library is ready for voting.

If there are any frequent Phobos contributors / core developers
please pay extra attention to submission code style and fitting
into overall Phobos guidelines and structure.

Most important goal of this review is to determine any API / 
design problems. Any internal implementation tweaks may happen 
after inclusion to Phobos but it is important to assure that no 
breaking changes will be required any time soon after module will 
get wider usage.

---- Information request from module author ----

Performance was a major discussed topic in previous thread. Could 
you please provide benchmarking data for version currently 
ongoing the review?