Use SIMD to accelerate comment lexing

[deadalnix via Digitalmars-d]

On Tuesday, 2 June 2015 at 12:27:38 UTC, Manu wrote:
> On 2 June 2015 at 05:39, Walter Bright via Digitalmars-d
> <digitalmars-d at puremagic.com> wrote:
>> https://issues.dlang.org/show_bug.cgi?id=14641
>>
>> Manu, our resident god of vector instructions, do you want to 
>> take this on?
>
> How do you measure this? Is there a convenient setup that will 
> produce
> a realistic test environment?
> This is more awkward in C than in D, it needs a different
> implementation for each compiler... will DMD's CI build with all
> common C compilers to prove that the implementations are 
> correct?

Well, I discussed that with clang people a while ago and here are 
how they do it and their measurement :

You go though character and look for a '/'. When you hit one, you 
check if the character before it is a *, and if so, you have the 
end of the comment. There is obviously various edges cases to 
take into account, but that is the general idea.

You can find the code in Lexer::SkipBlockComment in 
clang/lib/Lex/Lexer.cpp

Various benchmark on their side have shown that alignment is 
desirable before having vector operations to kick in. They used 
to have an AVX implementation, but it seems to be gone now, I'm 
not sure why at this stage.