std.d.lexer requirements

[Walter Bright]

On 8/2/2012 1:38 AM, Jonathan M Davis wrote:
> On Thursday, August 02, 2012 01:14:30 Walter Bright wrote:
>> On 8/2/2012 12:43 AM, Jonathan M Davis wrote:
>>> It is for ranges in general. In the general case, a range of UTF-8 or
>>> UTF-16 makes no sense whatsoever. Having range-based functions which
>>> understand the encodings and optimize accordingly can be very beneficial
>>> (which happens with strings but can't happen with general ranges without
>>> the concept of a variably-length encoded range like we have with forward
>>> range or random access range), but to actually have a range of UTF-8 or
>>> UTF-16 just wouldn't work. Range-based functions operate on elements, and
>>> doing stuff like filter or map or reduce on code units doesn't make any
>>> sense at all.
>>
>> Yes, it can work.
>
> How?

Keep a 6 character buffer in your consumer. If you read a char with the high bit 
set, start filling that buffer and then decode it.


> Do you really think that it makes sense for a function like map or filter to
> operate on individual code units? Because that's what would end up happening
> with a range of code units. Your average, range-based function only makes
> sense with _characters_, not code units. Functions which can operate on ranges
> of code units without screwing up the encoding are a rarity.

Rare or not, they are certainly possible, and the early versions of std.string 
did just that (although they weren't using ranges, the same techniques apply).