std.d.lexer requirements

[Andrei Alexandrescu]

On 8/2/12 6:54 PM, Jonathan M Davis wrote:
> So, a function which does the buffering of code units like Walter suggests is
> generic?

Of course, because it operates on bytes read from memory, files, or 
sockets etc.

> It's doing something that makes no sense outside of strings.

Right. The bytes represent UTF-8 encoded strings, except their type is 
ubyte so there's no processing in the library.

> And if
> it's doing that with strings and something else with everything else (which it
> _has_ to do if the same function is going to work with both unicode as well as
> range types that have nothing to do with unicode), then it's special casing
> strings and therefore is _not_ generic.

This is automatically destroyed because its assumption was destroyed.

> Sure, you could have a function which specifically operates on ranges of code
> units and understands how unicode works and is written accordingly, but then
> that function is specific to ranges of code units and is only generic with
> regards to various ranges of code units. It can't operate on generic ranges
> like functions such as map and filter can.

Yes, and I think that's exactly what the doctor prescribed here.


Andrei