Looking for champion - std.lang.d.lex

[Nick Sabalausky]

"Andrei Alexandrescu" <SeeWebsiteForEmail at erdani.org> wrote in message 
news:i9spsa$ll0$1 at digitalmars.com...
> On 10/22/10 14:02 CDT, Tomek Sowinski wrote:
>> Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 at digitalmars.com>
>> napisal(a):
>>
>>> As we all know, tool support is important for D's success. Making
>>> tools easier to build will help with that.
>>>
>>> To that end, I think we need a lexer for the standard library -
>>> std.lang.d.lex. It would be helpful in writing color syntax
>>> highlighting filters, pretty printers, repl, doc generators, static
>>> analyzers, and even D compilers.
>>>
>>
>> Interesting idea. Here's another: D will soon need bindings for CORBA,
>> Thrift, etc, so lexers will have to be written all over to grok
>> interface files. Perhaps a generic tokenizer which can be parametrized
>> with a lexical grammar would bring more ROI, I got a hunch D's templates
>> are strong enough to pull this off without any source code generation
>> ala JavaCC. The books I read on compilers say tokenization is a solved
>> problem, so the theory part on what a good abstraction should be is
>> done. What you think?
>
> Yes. IMHO writing a D tokenizer is a wasted effort. We need a tokenizer 
> generator.
>

FWIW, I've been converting my Goldie lexing/parsing library/toolset ( 
http://www.dsource.org/projects/goldie ) to D2/Phobos, and that should have 
a release sometime in the next couple months or so.

I'm not sure it would really be appropriate for Phobos since it's pretty 
big, comes with a bunch of tools (one of which is built in C# ATM), it's not 
range-ified yet, probably doesn't use Phobos coding conventions, and relies 
on one of my other libraries/tools.

But it does do generalized lexing/parsing (LALR) via the GOLD ( 
http://www.devincook.com/goldparser/ ) grammar file formats, can optionally 
generate source files for better compile-time checking (for instance, so 
Token!"<Statemnt>" will generate a compile-time error), has full 
documentation, and I'm working on a tool/lib that will compile the grammars 
without having to use the Windows/GUI-based GOLD Parser Builder tool.