std.d.lexer requirements

[Philippe Sigaud]

On Mon, Aug 6, 2012 at 8:03 PM, deadalnix <deadalnix at gmail.com> wrote:

> The most complex thing that is needed is the policy to allocate identifiers
> in tokens. It can be made by passing a function that have a string as
> parameter and a string as return value. The default one would be an identity
> function.

I think one should pass it an empty symbol table and the lexer should
fill it and associate each identifier with a unique ID, ID which would
appear in the Identifier token.


> The second parameter is a bool to tokenize comments or not. Is that enough ?
>
> The onError look like a typical use case for conditions as explained in the
> huge thread on Exception.

Yes, well we don't have a condition system. And using exceptions
during lexing would most probably kill its efficiency.
Errors in lexing are not uncommon. The usual D idiom of having an enum
StopOnError { no, yes } should be enough.