std.d.lexer : voting thread

[ilya-stromberg]

On Wednesday, 2 October 2013 at 14:41:56 UTC, Dicebot wrote:
> After brief discussion with Brian and gathering data from the 
> review thread, I have decided to start voting for `std.d.lexer` 
> inclusion into Phobos.

No.

I really want to see `std.d.lexer` in Phobos, but have too many 
conditions.

Documentation issues:

- please specify the parser algorithm that you used for 
`std.d.lexer`. As I understand from review thread, you implement 
`GLR parser` - please document it (correct me if I wrong). Also, 
add link to the algorithm description, for example to the 
wikipedia:
http://en.wikipedia.org/wiki/GLR_parser
It helps to understand how `std.d.lexer` works.
Also, please add best-case and worst-case time complexity (for 
example, from O(n) to O(n^3)), and best-case and worst-case 
memory complexity.

- please add more usage examples. Currently you have only one big 
example how generate HTML markup of D code. Try to add a simple 
example for every function.

- explicitly specify functions that can throw: add `Throws:` 
block for it and specify conditions when they can throw.

UTF-16/UTF-32 support:
- why standart `std.d.lexer` supports only UTF-8, but not a 
UTF-16/UTF-32? The official lexing specification allows all of 
them. The conversion from UTF-16/UTF-32 to UTF-8 is not a option 
due performance issues.
If Phobos string functions too slow, please add a bug. If Phobos 
haven't got necessary functions, please add enhancement request.
I think it's serious issue that affects all string utilities 
(like std.xml or std.json), not only `std.d.lexer`.

Exception handling
- please use `ParseException` as a default exception, not the 
`Exception`.

Codestyle:
- I don't like `TokenType` enum. You can use Tok!">>=" and 
`static if` to compare the token string to the `TokenType` enum. 
So, you will not lose performance, because string parsing will be 
done at compile time.

Not a condition, but wishlist:
- implement low-level API, not only high-level range-based API. I 
hope it can help increase performance for applications that 
really need it.

- add ability to use `std.d.lexer` at the compile time.