std.d.lexer: pre-voting review / discussion

[Timon Gehr]

On 09/12/2013 10:03 PM, Martin Nowak wrote:
> On 09/12/2013 05:39 PM, Timon Gehr wrote:
>>
>> Lookahead is realized as follows in the parser:
>>
>> (assume 'code' is the circular buffer range.)
>>
>> auto saveState(){muteerr++; return code.pushAnchor();} // saves the
>> state and mutes all error messages until the state is restored
>>
>> void restoreState(Anchor state){ muteerr--; code.popAnchor(state); }
>>
>> The 'Anchor' is a trivial wrapper around a size_t. The circular buffer
>> grows automatically to keep around tokens still reachable by an anchor.
>> (The range only needs small constant space besides the buffer to support
>> this functionality, though it is unable to detect usage errors.)
>
> Do you think it's possible to use CircularBufferRange itself as anchor.

Well, this implementation is geared towards the usage pattern found in 
the top-down parser. If the first anchor is not restored last, it will 
not generally work. This does not conform to the range interface. 
Implementing .save requires a more advanced and less efficient 
implementation.

> Then calling save() would return a CircularBufferRange and you could
> scratch the two functions above. I had some promising experiments in
> that direction,

What did you do? The way I think I'd approach it would be to maintain a 
binary min-heap using a few pointers in each range instance.

> but the implicit save on copy is annoying because you
> end up with anchors from temporary copies.

Is this referring to suboptimal efficiency? I guess that is rather hard 
to address in a safe way. Basically, you'd want to forget about "anchor 
management" in case it can be proven that there is another range on the 
same buffer that stays at most as progressed during the whole lifetime 
of the range under consideration.

Of course, it is always possible to do it in unsafely.