std.d.lexer: pre-voting review / discussion

[Walter Bright]

On 9/11/2013 6:54 PM, deadalnix wrote:
> On Thursday, 12 September 2013 at 01:39:52 UTC, Walter Bright wrote:
>> On 9/11/2013 6:30 PM, deadalnix wrote:
>>> Indeed. What solution do you have in mind ?
>>
>> The solution dmd uses is to put in an intermediary layer that saves the
>> lookahead tokens in a linked list.
>
> But then, you have an extra step when looking up every tokens + memory
> management overhead. How big is the performance improvement ?

Not really - I use a dirty trick in that a static instance is always the start 
of the list.

But even so, an extra indirection is better than re-lexing. Lexing is a clear 
bottleneck in the profiling. I've even been thinking of special casing the code 
that scans comments to use SIMD instructions.