Request for comments: std.d.lexer

[Jacob Carlborg]

On 2013-01-27 21:15, Philippe Sigaud wrote:

>> This means, for example, you'll need to squeeze pretty much all storage
>> allocation out of it. A lexer that does an allocation per token will is not
>> going to do very well at all.
>
> How does one do that? Honest question, I'm not really concerned by
> extreme speed, most of the time, and have lots to learn in this area.

If we're talking about dynamic allocation you can make sure you're just 
using value types. A token could look like:

struct Token
{
     TokenKind kind;
     string value;
     uint index;
}

For the "value" you could just slice the buffer. But I think this will 
prevent the whole buffer from being collected.

-- 
/Jacob Carlborg