DMD Compiler - lexer

[Ary Borenszweig via Digitalmars-d-learn]

On 8/29/14, 10:41 AM, Mike James wrote:
> Hi,
>
> Looking at the DMD Source Guide it says "The lexer transforms the file
> into an array of tokens."
>
> Why is this step taken instead of, say, just calling a function that
> returns the next token (or however many required for the look-ahead)?
>
> Regards,
>    -=mike=-

I believe this is just an abstract description of how it works. It 
actually uses something like next token with a freelist of tokens if it 
needs to do some lookahead.

https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L260
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L261
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L237