[Issue 14641] Use SIMD to accelerate comment lexing

[via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=14641

briancschott at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |briancschott at gmail.com

--- Comment #1 from briancschott at gmail.com ---
The best way to do this that I've found is to skip everything other than a set
of bytes that varies based on the comment being lexed:

For /* */ comments:
0x0c (\n)
0x0d (\r)
0x2a (*)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)

For /+ +/ comments:
0x0c (\n)
0x0d (\r)
0x2b (+)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)

For // comments:
0x0c (\n)
0x0d (\r)
x0e2 (Beginning of multi-byte UTF-8 newline)

The instruction used in libdparse to do this is "pcmpestri", which requires
SSE4.2 (First released in 2008 according to wikipedia). My advice is to leave
most of the logic intact and implement the assembly code such that it may
advance the lexer 0 or more bytes, so that the rest of the algorithm is not
disrupted on machines that don't support SSE4.2.

--