dmdz

[Andrei Alexandrescu]

On 03/17/2010 08:17 PM, Ellery Newcomer wrote:
> On 03/17/2010 03:53 PM, Andrei Alexandrescu wrote:
>> Thanks for replying to this. I'd been afraid that I was coming off too
>> critical. (I counted the ANTLR files as modules, and I think that's
>> fair.) To give you an idea on where I come from, distributing dmdz with
>> dmd is also a message to users on how things are getting done in D.
>
> dang right you are. If you're going to count the antlr runtime, then
> maybe you should also be counting druntime and the sections of phobos
> that I used?

I meant the antlr grammar for the task. I gave two counts, one excluding 
the antlr runtime, and based the rest of my discussion on that. I sadly 
note the irony. There is no need to get defensive, really.

>> For the problem "Compile a D file and all its dependents, link, and run"
>> the solution rdmd has 469 lines. It seems quite much to me, but I
>> couldn't find ways to make it much smaller.
>
> user wouldn't know that from any dmd distribution I've ever seen.
>
>>
>> For the problem "Given a zip file containing a D program, build it" the
>> dmdz solution is quite large. If we count everything:
>>
>> $ wc --lines dmdz.d import/antlrrt/*.d lexd.g opts.d sed.sh
>> 782 dmdz.d
>> 891 import/antlrrt/collections.d
>> 551 import/antlrrt/exceptions.d
>> 1253 import/antlrrt/lexing.d
>> 2085 import/antlrrt/parsing.d
>> 10 import/antlrrt/runtime.d
>> 600 import/antlrrt/utils.d
>> 436 lexd.g
>> 88 opts.d
>> 13 sed.sh
>> 6709 total
>
> forgot generated/*.d

Well that's generated. I counted what's needed to get things going. 
Unless you meant that ironically...

> should bump it up to 11 or 12 k.
>
>>
>> Arguably we can discount the import stuff, although I'd already raise
>> some objections:
>>
>> $ wc --lines dmdz.d lexd.g opts.d sed.sh
>> 782 dmdz.d
>> 436 lexd.g
>> 88 opts.d
>> 13 sed.sh
>> 1319 total
>>
>
> lexd.g and sed.sh are only there for reference. I hate it when machine
> generated source code is in a project, but the source grammar isn't.

My understanding is that lexd.g is your code so it should be included in 
the size of the solution, whereas the generated code should not.

>> That would suggest that it's about three times as difficult to build
>> stuff present in a zip file than to deduce dependencies and build stuff
>> not in a zip file. I find that difficult to swallow because to me
>> building stuff in a zip file should be in some ways easier because there
>> are no dependencies to deduce - they can be assumed to be in the zip
>> file.
>>
>> I looked more through the program and it looks like it uses the zip
>> library (honestly I would have used system("unzip...")), which does add
>> some aggravation for arguably a good reason. (But I also see there's no
>> caching, which is an important requirement.)
>
> eh?

The idea is to not extract the files every time you build. If they are 
in place already, the tool should recognize that.

>> Nice, but I don't know why you need to understand dmd's flags instead of
>> simply forwarding them to dmd. You could define dmdz-specific flags
>> which you parse and understand, and then dump everything else to dmd,
>> which will figure its own checking and error messages and all that.
>>
>
> filtering out flags that screw things up for the build in question;
> knowing where the resultant executable is supposed to be;
>
>>
>> I think it would be great to remove all stuff that's not necessary. I
>> paste at the end of this message my two baselines: a shell script and a
>> D program. They compare poorly with your program, but are extremely
>> simple. I think it may be useful to see how much impact each feature
>> that these programs lack is adding size to your solution.
>>
>>
>> Andrei
>
> You come at this problem like "It should be an eloquent showcase of what
> D has to offer."
>
> I come at it like "I want this to be generally useful. To me."

The tool shouldn't be a showcase. Obviously the primary purpose is for 
the tool to be useful. The shell script and the D script are useful. I 
am sure your tool is useful, but I think it doesn't hit the right 
balance. I simply don't think it takes that much code to achieve what 
the tool needs to achieve.

> In my opinion, how well it works trumps how many lines of code it took
> to write. But for the aforementioned bug, I never would have looked at
> rdmd's source, and even then I didn't notice how many lines of code it
> was. The way dmdz was written is based on the needs that presented
> themselves to me at the time. So far I've run it against three different
> projects and I'm happy with it the way it's turned out.
>
> 1. dmdz
>
> toy example. not much here.
>
> 2. dexcelapi
>
> port of jexcelapi, 90k loc (that thing must have shrunk when I wasn't
> looking, I was sure it was 200k), ~ 400 source files. Big. Dumping
> everything to dmd is easy enough to implement one way or another, but
> when I hit an Out of Memory Error I need what -piecemeal has to offer. I
> found the offending file (still don't know what's up with it), commented
> it out, and I can dump everything to dmd again. Without it, I probably
> would have given up on D for another year and a half.
>
> 3. dcrypt
>
> Today, I wanted to play with it, so I checked it out, popped dmdz.conf
> and a main.d in the directory and zipped the whole thing up.
>
> dmdz dcrypt.zip
>
> It worked. Without me doing anything to dmdz or dcrypt (except adding a
> string alias, &&^%^ tango).

I'm not contending the tool is not useful. I'm just saying it is too big 
for what it does, and that that does matter with regard to distributing 
it with dmd.

> I was kind of hoping others would try it and give their opinions, but
> apparently nobody else cares. Or they're on vacation, like I should be.
> Or they're giving the infamous 'silent approval'. Who knows.

It looks like we're getting into a little diatribe, which is very sad 
because you've clearly done a good amount of work and I didn't intend to 
make it look any other way. All I can say is that the tool is very far 
removed from what I think it should look like; for my money, the moment 
it gets larger than one simple module it would mean I took a few wrong 
turns along the way.

BTW Walter made a very nice suggestion: make a .zip file in the command 
line be equivalent to listing all files in that zip in the command line. 
I think it's this kind of idea that greatly simplifies things.


Andrei