Use SIMD to accelerate comment lexing

[Walter Bright via Digitalmars-d]

On 6/2/2015 5:45 PM, deadalnix wrote:
> Well, I discussed that with clang people a while ago and here are how they do it
> and their measurement :
>
> You go though character and look for a '/'. When you hit one, you check if the
> character before it is a *, and if so, you have the end of the comment. There is
> obviously various edges cases to take into account, but that is the general idea.
>
> You can find the code in Lexer::SkipBlockComment in clang/lib/Lex/Lexer.cpp
>
> Various benchmark on their side have shown that alignment is desirable before
> having vector operations to kick in. They used to have an AVX implementation,
> but it seems to be gone now, I'm not sure why at this stage.

Line numbers have to be kept track of as well.