std.d.lexer requirements

Piotr Szturmaj bncrbme at jadamspam.pl
Thu Aug 2 02:27:14 PDT 2012


Walter Bright wrote:
> 1. It should accept as input an input range of UTF8. I feel it is a
> mistake to templatize it for UTF16 and UTF32. Anyone desiring to feed it
> UTF16 should use an 'adapter' range to convert the input to UTF8. (This
> is what component programming is all about.)

Why it is a mistake? I think Lexer should parse any UTF range and return 
compatible token's strings. That is it should provide strings for UTF8 
input, wstrings for UTF16 input and so on.


More information about the Digitalmars-d mailing list