What's the simplest way to read a file token by token?

[Tobias Pankrath]

On 10.08.2013 19:34, Carl Sturtivant wrote:
> On Saturday, 10 August 2013 at 17:09:29 UTC, Carl Sturtivant wrote:
>>
>> What's the simplest way in D to read a file token by token, where the
>> read tokens are D strings, and they are separated in the file by
>> arbitrary non-zero amounts of white space (including spaces, tabs and
>> newlines at a minimum)?
>
> I couldn't find a function that did just this, and various ways I
> implemented it seemed too complex. Is there such a function in a D library?

There are some candidates for std.d.lexer on the way. Try for example:
https://github.com/Hackerpilot/Dscanner/blob/master/stdx/d/lexer.d