Lexer in D

[Jonathan M Davis]

On Sunday, March 03, 2013 17:19:25 Namespace wrote:
> But hasn't each range an array internally?

Many do, but some don't. Take std.algorithm.ioto, or the ranges in std.random 
for instance. They're generative. And if all ranges had arrays underneat the 
hood, infinite ranges wouldn't be possible except for something like 
std.algorithm.cycle. Something like

struct Range
{
    @property int front() { return _i; }
    void popFront() {++i}
    @property bool empty { return _i != int.max; }

    private int _i;
}

would be a valid range. In the case of a lexer, you generate the next token on 
each popFront call, so you consume the range of characters that you're lexing 
as you go but don't need to allocate memory for more than the current token at 
a time.

- Jonathan M Davis