std.d.lexer: pre-voting review / discussion

Dmitry Olshansky dmitry.olsh at gmail.com
Thu Sep 12 09:21:19 PDT 2013


12-Sep-2013 12:05, Jacob Carlborg пишет:
> On 2013-09-11 17:01, Dicebot wrote:
>> std.d.lexer is standard module for lexing D code, written by Brian Schott
>
> Finally :)
>
> * How does it handler errors, just returns TokenType.invalid?
>
> * Personally I think the module is too big. I would go with:
>
> - std.d.lexer.token
> - std.d.lexer.tokentype

These could be one module. There is really no meaningful way to use 
token type separately from token.

> - std.d.lexer.lexer - contains the rest
> - std.d.lexer.config - IterationStyle, TokenStyle, LexerConfig

Contrary I see this break down pointless - do you really want to use 
config without the lexer?


> - CircularRange, StringCache, possibly put somewhere else. I assume this
> can be used for other things than lexing?
> - Trie related code, same as above

No good public interface defined is the reason. Basically the same as 
with Trie in the new std.uni module - needs its own review.

>
> * I see that errorMessage throws an exception. Do we really want that? I
> would except it just returns an invalid token.
>
> If we do decide it should throw, it should absolutely _not_ throw a
> plain Exception. Create a new type, LexException or similar. I hate when
> code throws plain Exceptions, it makes it useless to catch them.
>
> I would also expect this LexException to contain a Token. It shouldn't
> be needed to parse the exception message to get line and column
> information.

Better yet to have a std exception hierarchy... so that all parsing 
modules can be tied to ParseException. So this needs to be resolved in a 
forward-compatible way.


-- 
Dmitry Olshansky


More information about the Digitalmars-d mailing list