Looking for champion - std.lang.d.lex

[Jonathan M Davis]

On Friday, November 19, 2010 15:17:35 Bruno Medeiros wrote:
> On 19/11/2010 22:02, Jonathan M Davis wrote:
> > On Friday, November 19, 2010 13:53:12 Bruno Medeiros wrote:
> >> On 19/11/2010 21:27, Jonathan M Davis wrote:
> >> 
> >> And by providing a lexer and a parser outside the standard library,
> >> wouldn't it make it just as easy for those tools to be written? What's
> >> the advantage of being in the standard library? I see only
> >> disadvantages: to begin with it potentially increases the time that
> >> Walter or other Phobos contributors may have to spend on it, even if
> >> it's just reviewing patches or making sure the code works.
> > 
> > If nothing, else, it makes it easier to keep in line with dmd itself.
> > Since the dmd front end is LGPL, it's not possible to have a Boost port
> > of it (like the Phobos version will be) without Walter's consent. And
> > I'd be surprised if he did that for a third party library (though he
> > seems to be pretty open on a lot of that kind of stuff). Not to mention,
> > Walter and the core developers are _exactly_ the kind of people that you
> > want working on a lexer or parser of the language itself, because
> > they're the ones who work on it.
> > 
> > - Jonathan M Davis
> 
> Eh? That license argument doesn't make sense: if the lexer and parser
> were to be based on DMD itself, then putting it in the standard library
> is equivalent (in licensing terms) to licensing the lexer and parser
> parts of DMD in Boost. More correctly, what I mean by equivalent, is
> that there no reason why Walter would allow one thing and not the
> other... (because on both cases he would have to issue that license)

It's very different to have D implementation of something - which is based on a 
C++ version but definitely different in some respects - be under Boost and 
generally available, and having the C++ implementation be under Boost - 
particularly when the C++ version covers far more than just a lexer and parser. 
Someone _could_ port the D code back to C++ and have that portion useable under 
Boost, but that's a lot more work than just taking the C++ code and using it, 
and it's only the portions of the compiler which were ported to D to which could 
be re-used that way. And since the Boost code could be used in a commercial 
product while the LGPL is more restricted, it could make a definite difference.

I'm not a licensing expert, and I'm not an expert on what Walter does and 
doesn't want done with his code, but he put the compiler front end under the 
LGPL, not Boost, and he's given his permission to have the lexer alone ported to 
D and put under the Boost license in the standard library, which is very 
different from putting the entire front end under Boost. I expect that the parser 
will follow eventually, but even if it does, that's still not the entire front 
end. So, there is a difference in licenses does have a real impact. And no one 
can take the LGPL C++ code and port it to D - for the standard library or 
otherwise - without Walter's permission, because its his copyright on the code.

As for the usefulness of a D lexer and parser, I've already had several programs 
or functions which I've wanted to write which would require it, and the lack has 
made them infeasible. For instance, I was considering posting a version of my 
datetime code without the unit tests in it, so that it would be easier to read 
the actual code (given the large number of unit tests), but I found that to 
accurately do that, you need a lexer for D, so I gave up on it for the time 
being. Having a function which stripped out unnecessary whitespace (and 
especially newlines) for string mixins would be great (particularly since line 
numbers get messed up with multi-line string mixins), but that would require  
CTFE-able D lexer to work correctly (though you might be able to hack together 
something which would mostly work), which we don't have. The D lexer won't be 
CTFE-able initially (though hopefully it will be once the CTFE capabilites of 
dmd improve), so you still won't be able to do that once the lexer is done, but 
it is a case where a lexer would be useful.

Ideally, Phobos would be huge in a manner similar to how C# or Java's libraries 
are huge. It will take time to get there, and we'll need more developers, but I 
don't think that it really makes sense to not put things in the standard library 
because it might take more dev time - particularly when a D lexer is the sort of 
thing that likely won't need much changing once it's done, since it would only 
need to be changed when the language changed or when a bug with it was found 
(which would likely equate to a bug in the compiler anyway), so ultimately, the 
developer cost is likely fairly low. Additionally, Walter thinks that the 
development costs will be lower to have it in the standard library with an 
implementation similar to dmd's rather than having it separate. And it's his 
call. So, it's going to get done. There are several people around here who 
lament the lack of D parser in Phobos at least periodically, and I think that it 
will be good to have an appropriate lexer and parser for D in Phobos. Having 
other 3rd party stuff - like antlr - is great too, but that's no reason not to 
put it in the standard library.

I think that we're just going to have to agree to disagree on this one.

- Jonathan M Davis