Looking for champion - std.lang.d.lex

[Tomek Sowiński]

Dnia 22-10-2010 o 00:01:21 Walter Bright <newshound2 at digitalmars.com>  
napisał(a):

> As we all know, tool support is important for D's success. Making tools  
> easier to build will help with that.
>
> To that end, I think we need a lexer for the standard library -  
> std.lang.d.lex. It would be helpful in writing color syntax highlighting  
> filters, pretty printers, repl, doc generators, static analyzers, and  
> even D compilers.
>
> It should:
>
> 1. support a range interface for its input, and a range interface for  
> its output
> 2. optionally not generate lexical errors, but just try to recover and  
> continue
> 3. optionally return comments and ddoc comments as tokens
> 4. the tokens should be a value type, not a reference type
> 5. generally follow along with the C++ one so that they can be  
> maintained in tandem
>
> It can also serve as the basis for creating a javascript implementation  
> that can be embedded into web pages for syntax highlighting, and  
> eventually an std.lang.d.parse.
>
> Anyone want to own this?

Interesting idea. Here's another: D will soon need bindings for CORBA,  
Thrift, etc, so lexers will have to be written all over to grok interface  
files. Perhaps a generic tokenizer which can be parametrized with a  
lexical grammar would bring more ROI, I got a hunch D's templates are  
strong enough to pull this off without any source code generation ala  
JavaCC. The books I read on compilers say tokenization is a solved  
problem, so the theory part on what a good abstraction should be is done.  
What you think?

--
Tomek