Request for comments: std.d.lexer

[Philippe Sigaud]

On Sun, Jan 27, 2013 at 8:48 PM, Walter Bright
<newshound2 at digitalmars.com> wrote:
> On 1/27/2013 2:17 AM, Philippe Sigaud wrote:
>>
>> Walter seems to think if a lexer is not able to vomit thousands
>> of tokens a seconds, then it's not good.
>
>
> Speed is critical for a lexer.

Something I never tought about: given a 10k lines module, how many
tokens does that represent, roughly. Ten times as much?

> This means, for example, you'll need to squeeze pretty much all storage
> allocation out of it. A lexer that does an allocation per token will is not
> going to do very well at all.

How does one do that? Honest question, I'm not really concerned by
extreme speed, most of the time, and have lots to learn in this area.