std.d.lexer requirements

[Walter Bright]

On 8/2/2012 12:21 AM, Jonathan M Davis wrote:
>> Because your input range is a range of dchar?
> I think that we're misunderstanding each other here. A typical, well-written,
> range-based function which operates on ranges of dchar will use static if or
> overloads to special-case strings. This means that it will function with any
> range of dchar, but it _also_ will be as efficient with strings as if it just
> operated on strings.

It *still* must convert UTF8 to dchars before presenting them to the consumer of 
the dchar elements.


> It won't decode anything in the string unless it has to.
> So, having a lexer which operates on ranges of dchar does _not_ make string
> processing less efficient. It just makes it so that it can _also_ operate on
> ranges of dchar which aren't strings.
>
> For instance, my lexer uses this whenever it needs to get at the first
> character in the range:
>
> static if(isNarrowString!R)
>      Unqual!(ElementEncodingType!R) first = range[0];
> else
>      dchar first = range.front;

You're requiring a random access input range that has random access to something 
other than the range element type?? and you're requiring an isNarrowString to 
work on an arbitrary range?


> if I need to know the number of code units that make up the code point, I
> explicitly call decode in the case of a narrow string. In either case, code
> units are _not_ being converted to dchar unless they absolutely have to be.

Or you could do away with requiring a special range type and just have it be a 
UTF8 range.

What I wasn't realizing earlier was that you were positing a range type that has 
two different kinds of elements. I don't think this is a proper component type.


> Yes. I understand. It has a mapping of pointers to identifiers. My point is
> that nothing but parsers will need that.
> From the standpoint of functionality,
> it's a parser feature, not a lexer feature. So, if it can be done just fine in
> the parser, then that's where it should be. If on the other hand, it _needs_
> to be in the lexer for some reason (e.g. performance), then that's a reason to
> put it there.

If you take it out of the lexer, then:

1. the lexer must allocate storage for every identifier, rather than only for 
unique identifiers

2. and then the parser must scan the identifier string *again*

3. there must be two hash lookups of each identifier rather than one

It's a suboptimal design.