std.d.lexer requirements

[Andrei Alexandrescu]

On 8/2/12 7:18 PM, Jonathan M Davis wrote:
> On Thursday, August 02, 2012 19:06:32 Andrei Alexandrescu wrote:
>>> Sure, you could have a function which specifically operates on ranges of
>>> code units and understands how unicode works and is written accordingly,
>>> but then that function is specific to ranges of code units and is only
>>> generic with regards to various ranges of code units. It can't operate on
>>> generic ranges like functions such as map and filter can.
>>
>> Yes, and I think that's exactly what the doctor prescribed here.
>
> It may be the best approach for the lexer (though I'm not convinced; I'll have
> to think about it more),

Your insights are always appreciated; even their Cliff notes :o).

> but Walter seems to be arguing that that strings
> should be treated as ranges of code units in general, which I think is
> completely wrong.

I think Walter has very often emphasized the need for the lexer to be 
faster than the usual client software. My perception is that he's 
discussing lexer design in understanding there's a need for a less 
comfortable approach, namely do decoding in client.

Andrei