std.d.lexer : voting thread

[Andrei Alexandrescu]

On 10/6/13 2:10 AM, Jacob Carlborg wrote:
> On 2013-10-05 02:24, Andrei Alexandrescu wrote:
>
>> Such a trie searcher is not intelligent, but is very composable and
>> extremely fast. It is just smart enough to do maximum munch (e.g.
>> interprets "==" and "foreach" as one token each, not two), but is not
>> smart enough to distinguish an identifier "whileTrue" from the keyword
>> "while" (it claims "while" was found and stops right at the beginning of
>> "True" in the stream). This is for generality so applications can define
>> how identifiers work (e.g. Lisp allows "-" in identifiers but D doesn't
>> etc). The trie finder doesn't do numbers or comments either. No regexen
>> of any kind.
>
> Would it be able to lex Scala and Ruby? Method names in Scala can
> contain many symbols that is not usually allowed in other languages. You
> can have a method named "==". In Ruby method names are allowed to end
> with "=", "?" or "!".

Yes, easily. Have the trie matcher stop upon whatever symbol it detects 
and then handle the tail with Ruby-specific code.

Andrei