std.d.lexer requirements

[José Armando García Sancio]

On Thu, Aug 2, 2012 at 1:26 PM, Jonathan M Davis <jmdavisProg at gmx.com> wrote:
> On Thursday, August 02, 2012 01:44:18 Walter Bright wrote:
>> On 8/2/2012 1:38 AM, Jonathan M Davis wrote:
>> > On Thursday, August 02, 2012 01:14:30 Walter Bright wrote:
>> >> On 8/2/2012 12:43 AM, Jonathan M Davis wrote:
>> >>> It is for ranges in general. In the general case, a range of UTF-8 or
>> >>> UTF-16 makes no sense whatsoever. Having range-based functions which
>> >>> understand the encodings and optimize accordingly can be very beneficial
>> >>> (which happens with strings but can't happen with general ranges without
>> >>> the concept of a variably-length encoded range like we have with forward
>> >>> range or random access range), but to actually have a range of UTF-8 or
>> >>> UTF-16 just wouldn't work. Range-based functions operate on elements,
>> >>> and
>> >>> doing stuff like filter or map or reduce on code units doesn't make any
>> >>> sense at all.
>> >>
>> >> Yes, it can work.
>> >
>> > How?
>>
>> Keep a 6 character buffer in your consumer. If you read a char with the high
>> bit set, start filling that buffer and then decode it.
>
> And how on earth is that going to work as a range? Range-based functions
> operate on elements. They use empty, front, popFront, etc. If front doesn't
> return an element that a range-based function can operate on without caring
> what it is, then that type isn't going to work as a range. If you need the
> consumer to be doing something special, then that means you need to special
> case it for that range type. And that's what you're doing when you special-
> case range-base functions for strings.

A little bit off topic but...

People have been composing/decorating Streams/Ranges for probably 30
years now. Examples: input stream, output stream, byte stream, char
stream, buffered stream, cipher stream, base64 stream, etc.

If you need more example. Consider an HTTPS request. At the lowest
level you have a byte stream/range. No sane developer wants to deal
with HTTPS request at this level so you decorate it with an SSL
stream/range. That is still too low level so you decorate this with a
char stream/range. Still too low level? Decorate it with a modal line
buffered stream/range. We are getting closer but it still not the
correct range abstraction so then you need a modal http stream/range.
You need the modal part if you want to support http streaming.