DMD Compiler - lexer

[Mike James via Digitalmars-d-learn]

Hi,

Looking at the DMD Source Guide it says "The lexer transforms the 
file into an array of tokens."

Why is this step taken instead of, say, just calling a function 
that returns the next token (or however many required for the 
look-ahead)?

Regards,
   -=mike=-