DCT: D compiler as a collection of libraries

[Jonathan M Davis]

On Friday, May 11, 2012 10:01:28 Roman D. Boiko wrote:
> There were several discussions about the need for a D compiler
> library.
> 
> I propose my draft implementation of lexer for community review:
> https://github.com/roman-d-boiko/dct
> 
> Lexer is based on Brian Schott's project
> https://github.com/Hackerpilot/Dscanner, but it has been
> refactored and extended (and more changes are on the way).
> 
> The goal is to have source code loading, lexer, parser and
> semantic analysis available as parts of Phobos. These libraries
> should be designed to be usable in multiple scenarios (e.g.,
> refactoring, code analysis, etc.).
> 
> My commitment is to have at least front end built this year (and
> conforming to the D2 specification unless explicitly stated
> otherwise for some particular aspect).
> 
> Please post any feed here. A dedicated project web-site will be
> created later.

It's great that you're working on this. We definitely need more of this sort of 
stuff. However, I would point out that I'm not sure that it will be acceptable 
for Phobos (it _may_ be, but it's not quite what we've been looking for). 
There are two types of lexers/parsers that we've been looking to add to 
Phobos.

1. A direct port of the lexer from dmd's C++ front-end. The API would be D-
ified to use ranges, but the internals would be almost identical to the C++ 
front-end so that porting changes back and forth would be very easy. It also 
would make it easier to replace the C++ front-end with a D one later if the D 
one is very close to the C++ one.

2. A lexer/parser generator using templates and/or CTFE to generate a 
lexer/parser of whatever language using a BNF or EBNF grammar - probably 
similar to what Philippe Sigaud has done to generate a parser using PEGs. Such 
could be used to generate a lexer/parser for D or any other language.

What you seem to be doing is creating a D-specific parser from scratch, which 
is great, but it's not quite what we've been looking to add to Phobos. That 
doesn't necessarily mean that we can't or won't add it to Phobos once it's 
complete (especially if nothing else gets done first), but it also may not be 
acceptable, because it's not either #1 or #2. That would have to be decided 
once it's done and ready to be reviewed for inclusion in Phobos. I'm not 
trying to discourage you at all. I'm just pointing out that what you're doing 
is quite what we're looking for. Regardless, it _would_ be interesting to be 
able to compare implementations of #1 and #2 against what you're doing and 
seeing which performs better.

- Jonathan M Davis