Use SIMD to accelerate comment lexing

[deadalnix via Digitalmars-d]

On Wednesday, 3 June 2015 at 22:50:52 UTC, Walter Bright wrote:
> On 6/2/2015 5:45 PM, deadalnix wrote:
>> Well, I discussed that with clang people a while ago and here 
>> are how they do it
>> and their measurement :
>>
>> You go though character and look for a '/'. When you hit one, 
>> you check if the
>> character before it is a *, and if so, you have the end of the 
>> comment. There is
>> obviously various edges cases to take into account, but that 
>> is the general idea.
>>
>> You can find the code in Lexer::SkipBlockComment in 
>> clang/lib/Lex/Lexer.cpp
>>
>> Various benchmark on their side have shown that alignment is 
>> desirable before
>> having vector operations to kick in. They used to have an AVX 
>> implementation,
>> but it seems to be gone now, I'm not sure why at this stage.
>
> Line numbers have to be kept track of as well.

They retrieve line number lazily when needed, with various 
mechanism to speedup the lookup.