std.d.lexer : voting thread

[ilya-stromberg]

On Tuesday, 8 October 2013 at 00:16:45 UTC, Andrei Alexandrescu 
wrote:
> To put my money where my mouth is, I have a proof-of-concept 
> tokenizer for C++ in working state.
>
> http://dpaste.dzfl.pl/d07dd46d

Why do you use "\0" as end-of-stream token:

   /**
    * All token types include regular and reservedTokens, plus the 
null
    * token ("") and the end-of-stream token ("\0").
    */

We can have situation when the "\0" is a valid token, for example 
for binary formats. Is it possible to indicate end-of-stream 
another way, maybe via "empty" property for range-based API?