Request for comments: std.d.lexer

[Andrei Alexandrescu]

On 2/5/13 5:44 AM, Jonathan M Davis wrote:
> On Tuesday, February 05, 2013 09:14:35 Jacob Carlborg wrote:
>> On 2013-02-05 04:22, Andrei Alexandrescu wrote:
>>> Suggestion: take lexer.c and convert it to D. Should take one day, and
>>> you'll have performance on par.
>>
>> There's reason for why nobody has just extract the lexer from DMD. It
>> will probably take more than a day just to extract the lexer to be able
>> to use it without the rest of DMD.
>
> There are basic ideas about how it works which are obviously good and should
> be in the finished product in D, but it's not range-based, which forces you to
> do things differently. It's also not configurable, which forces you to do things
> differently.
>
> If it could be ported as-is and then compared for speed, then that would be a
> great test, since it would be able to show how much of the speed problem is
> purely a compiler issue as opposed to a design issue, but you wouldn't be able
> to actually use it for anything more than what Brian is doing with his
> performance testing, because as you point out, it's too integrated into dmd.
> It _would_ be valuable though as a performance test of the compiler.

As far as I could tell the dependencies of the lexer are fairly 
contained (util, token, identifier) and conversion to input range is 
immediate.

Andrei