std.d.lexer requirements

[Jonathan M Davis]

On Thursday, August 02, 2012 18:41:23 Andrei Alexandrescu wrote:
> On 8/2/12 6:38 PM, Jonathan M Davis wrote:
> > On Thursday, August 02, 2012 15:14:17 Walter Bright wrote:
> >> Remember, its the consumer doing the decoding, not the input range.
> > 
> > But that's the problem. The consumer has to treat character ranges
> > specially to make this work. It's not generic.
> 
> It is generic! It's just in another dimension: it operates on any range
> of _bytes_.

So, a function which does the buffering of code units like Walter suggests is 
generic? It's doing something that makes no sense outside of strings. And if 
it's doing that with strings and something else with everything else (which it 
_has_ to do if the same function is going to work with both unicode as well as 
range types that have nothing to do with unicode), then it's special casing 
strings and therefore is _not_ generic.

Sure, you could have a function which specifically operates on ranges of code 
units and understands how unicode works and is written accordingly, but then 
that function is specific to ranges of code units and is only generic with 
regards to various ranges of code units. It can't operate on generic ranges 
like functions such as map and filter can.

- Jonathan M Davis