DMD Compiler - lexer

[ketmar via Digitalmars-d-learn]

On Fri, 29 Aug 2014 13:41:20 +0000
Mike James via Digitalmars-d-learn <digitalmars-d-learn at puremagic.com>
wrote:

> Looking at the DMD Source Guide it says "The lexer transforms the 
> file into an array of tokens."
> 
> Why is this step taken instead of, say, just calling a function 
> that returns the next token (or however many required for the 
> look-ahead)?
D parser is somewhat complicated and it can peek alot of tokens from
input stream without consuming 'em. so it's just easier to convert
source code to tokens and then work with token stream.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20140829/bd21f321/attachment.sig>