std.d.lexer: pre-voting review / discussion

[Walter Bright]

On 9/11/2013 12:55 PM, H. S. Teoh wrote:
>> 3. I assumed TokenType is a type. But it's not, it's an enum. Even
>> the document says it's a 'type', but it's not a type.
>
> I don't understand this complaint. I thought it's pretty clear that it's
> referring to what type of token it is (or would you prefer TokenKind?).

A type is a well-understood concept in D. This is using 'type' to mean something 
completely different. This is very jarring. Your idea of using 'kind' is vastly 
superior.


>> 5. The LexerConfig initialization should be a constructor rather than
>> a sequence of assignments. LexerConfig documentation is awfully thin.
>> For example, 'tokenStyle' is explained as being 'Token style',
>> whatever that is.
>
> I'm on the fence about this one. Setting up the configuration before
> starting the lexing process makes sense to me.

Yes, using a constructor :-)

>> 6. No clue how lookahead works with this. Parsing D requires arbitrary
>> lookahead.
>
> What has this got to do with lexing?

The lexer needs to support backing up and going forward again. That's how the 
dmd lexer works.


> Also, it's unclear what types of input is supported -- the code example
> only uses string input, but what about file input? Does it support
> byLine? Or does it need me to slurp the entire file contents into an
> in-memory buffer first?

The point of having the input be an InputRange is you DO NOT CARE if the input 
is coming from a string, file, etc.