std.d.lexer: pre-voting review / discussion

[Timon Gehr]

On 09/11/2013 08:49 PM, Walter Bright wrote:
>

>
> 3. I assumed [an instance of] TokenType is a type.

(is(TokenType) is true).

> But it's not, it's an enum.
> Even the document says it's a 'type', but it's not a type.

It's a type tag. The tag uniquely determines the type. (As in 'the type 
of a token', as opposed to 'the type of an expression'.)

> 4. When naming tokens like .. 'slice', it is giving it a
> syntactic/semantic name rather than a token name. This would be awkward
> if .. took on new meanings in D. Calling it 'dotdot' would be clearer.
> Ditto for the rest. For example that is done better, '*' is called
> 'star', rather than 'dereference'.

FWIW, I use Tok!"..". I.e. a "UDL" for specifying kinds of tokens when 
interfacing with the parser. Some other kinds of tokens get a canonical 
representation. Eg. Tok!"i" is the kind of identifier tokens, Tok!"0" is 
the kind of signed integer literal tokens etc.

>
> 5. The LexerConfig initialization should be a constructor rather than a sequence of assignments.

Using the { a:2, b:3 }-style initialization syntax?

> 6. No clue how lookahead works with this.

Eg. use a CircularBuffer adapter range. I have an implementation 
currently coupled with my own lexer implementation. If there is 
interest, I could factor it out.

Lookahead is realized as follows in the parser:

(assume 'code' is the circular buffer range.)

auto saveState(){muteerr++; return code.pushAnchor();} // saves the 
state and mutes all error messages until the state is restored

void restoreState(Anchor state){ muteerr--; code.popAnchor(state); }

The 'Anchor' is a trivial wrapper around a size_t. The circular buffer 
grows automatically to keep around tokens still reachable by an anchor. 
(The range only needs small constant space besides the buffer to support 
this functionality, though it is unable to detect usage errors.)


This approach is typically more efficient than using a free list on 
contemporary architectures.