What's the simplest way to read a file token by token?

[Carl Sturtivant]

On Saturday, 10 August 2013 at 18:45:55 UTC, Tobias Pankrath 
wrote:
>
> There are some candidates for std.d.lexer on the way. Try for 
> example:
> https://github.com/Hackerpilot/Dscanner/blob/master/stdx/d/lexer.d

OK, but that seems to solve a more difficult problem: my tokens 
are all separated by non-empty white space. In other languages 
I've found a function that simply reads token by token in this 
simple situation.