Lexer in D

[Dmitry Olshansky]

03-Mar-2013 20:19, Namespace пишет:
>> Simple - don't use array append and don't produce and array.
>> Just produce a lazy forward range that is iterated.
>>
>> At the very least use Appender!(Token[]) it ought to be much faster.
>
> But hasn't each range an array internally?

Of course, not or the whole range concept is meaningless.

> Or how does that work?

By having a struct or generally any object that in fact works as 
generator of tokens.

popFront lexes new token, front gives the current token and empty 
indicate if there are no more tokens. See std.range for some simple ranges.

If you throw in a .save method you can replay tokenization from some 
point in the past (= soem older state, think lookahead when parsing).

> I try to use Appender, but this lazy forward range sounds interesting.


-- 
Dmitry Olshansky