std.d.lexer: pre-voting review / discussion

[Dominikus Dittes Scherkl]

On Thursday, 26 September 2013 at 16:47:09 UTC, Jos van Uden 
wrote:
>> Is U+001A really meant to end the source file?
>> According to the Unicode specification this is a "replacement 
>> character", like the newer U+FFFC. Or is it simply a spelling 
>> error and U+0019 was intended to
>> end the source (this would fit, as it means "end of media").
>>
>> More important to me is, that all the Space-Characters beyond 
>> ASCII are not considered whitespace
>
> I imagine the lexer follows the language specification:
>
> http://dlang.org/lex.html#EndOfFile

I know. What I wanted to say is: The language specification has a 
bug here
(at least it is strange to interpret "replacement character" as 
end of file
and "end of media" not) and the handling of unicode space 
characters is
not nice.
If this is not the right place to discus that matter, please 
point me to a better place.