Lexer in D

[Namespace]

On Sunday, 3 March 2013 at 16:19:27 UTC, Namespace wrote:
>> Simple - don't use array append and don't produce and array.
>> Just produce a lazy forward range that is iterated.
>>
>> At the very least use Appender!(Token[]) it ought to be much 
>> faster.
>
> But hasn't each range an array internally?
> Or how does that work?
>
> I try to use Appender, but this lazy forward range sounds 
> interesting.

Appender does not work for my Token struct because I have 
const/immutable members.
So I cannot add something.

Example: http://dpaste.1azy.net/21f100d3
Without 'const' it works fine.