Lexer and parser generators using CTFE

[H. S. Teoh]

On Wed, Feb 29, 2012 at 07:28:48PM +0100, Martin Nowak wrote:
> On Wed, 29 Feb 2012 17:41:19 +0100, Timon Gehr <timon.gehr at gmx.ch> wrote:
> 
> >On 02/28/2012 07:46 PM, Martin Nowak wrote:
> >>
> >>https://gist.github.com/1255439 - lexer generator
> >>https://gist.github.com/1262321 - complete and fast D lexer
> >>
> >
> >Well, it is slower at lexing than DMD at parsing. What is the bottleneck?
> 
> No, it's as fast as dmd's lexer.
> 
> Writing the tokens to stdout takes a lot of time though.
> Just disable the "writeln(tok);" in the main loop.
> There is also an --echo option which recreates the complete
> source from the tokens.

This sounds like something I ran into in my D lexer project. Lexing
Phobos took upwards of 10-15 seconds, which is extremely slow
considering that dmd can *compile* it in just a few seconds. So I did
some profiling, and found out that most of the time was spent formatting
tokens into strings and running writeln. After I commented that out, the
running time dropped to just a few seconds. :-)


T

-- 
The fact that anyone still uses AOL shows that even the presence of
options doesn't stop some people from picking the pessimal one. - Mike
Ellis