std.d.lexer: pre-voting review / discussion

[Walter Bright]

On 9/11/2013 8:01 AM, Dicebot wrote:
> std.d.lexer is standard module for lexing D code, written by Brian Schott

Thank you, Brian! This is important work.

Not a thorough review, just some notes from reading the doc file:

1. I don't like the _ suffix for keywords. Just call it kwimport or something 
like that.

2. The example uses an if-then sequence of isBuiltType, isKeyword, etc. Should 
be an enum so a switch can be done for speed.

3. I assumed TokenType is a type. But it's not, it's an enum. Even the document 
says it's a 'type', but it's not a type.

4. When naming tokens like .. 'slice', it is giving it a syntactic/semantic name 
rather than a token name. This would be awkward if .. took on new meanings in D. 
Calling it 'dotdot' would be clearer. Ditto for the rest. For example that is 
done better, '*' is called 'star', rather than 'dereference'.

5. The LexerConfig initialization should be a constructor rather than a sequence 
of assignments. LexerConfig documentation is awfully thin. For example, 
'tokenStyle' is explained as being 'Token style', whatever that is.

6. No clue how lookahead works with this. Parsing D requires arbitrary lookahead.

7. uint line; Should indicate that lines start with '1', not '0'. Ditto for columns.

8. 'default_' Again with the awful practice of appending _.

9. Need to insert intra-page navigation links, such as when 'byToken()' appears 
in the text, it should be link to where byToken is described.


> Goal of this thread is to detect if there are any outstanding
> issues that need to fixed before formal "yes"/"no" voting
> happens. If no critical objections will arise, voting will begin
> starting with a next week. Otherwise it depends on time module author needs to implement suggestions.

I believe the state of the documentation is a showstopper, and needs to be 
extensively fleshed out before it can be considered ready for voting.