What's going on with std.experimental.lexer?

[Brian Schott via Digitalmars-d]

On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
> I've been looking at ways to optimize the D lexer's operation 
> using SIMD instructions. I'm not yet sure if I'll need to 
> change the lexer generator's API to do this. I'm going to wait 
> until I have my proof-of-concept code and some benchmark 
> results before asking for a voting thread or creating a pull 
> request.

Implementing some SIMD code just in the lexWhitespace function 
causes a drop in total lexing time of roughly 3.7%. This looks 
promising so far, so I'm going to implement similar code in 
lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and 
lexSlashPlusComment.

So far this only uses SSE2 instructions, so all 64-bit x86 
processors can benefit from this. (SSE4.2 would allow us to do 
more interesting things, but it's not as common)