Request for comments: std.d.lexer

[Mehrdad]

On Sunday, 27 January 2013 at 20:15:33 UTC, Philippe Sigaud wrote:

>> This means, for example, you'll need to squeeze pretty much 
>> all storage allocation out of it. A lexer that does an 
>> allocation per token will is not going to do very well at all.
>
> How does one do that? Honest question, I'm not really concerned 
> by extreme speed, most of the time, and have lots to learn in 
> this area.


Here's my (VERY) simple NFA-based "regex"-based lexer, which 
performs **no** heap allocations (unless the pattern is very 
complicated):

https://gist.github.com/b10ae22ab822c87467a3

Yes, the code is ugly and the capabilities are far too basic, but 
it was good enough for what I needed.

The point it illustrates is how you can lex without any heap 
allocations.