dmdz

[Andrei Alexandrescu]

On 03/17/2010 03:01 PM, Ellery Newcomer wrote:
> On 03/16/2010 08:13 PM, Andrei Alexandrescu wrote:
>> This is solid work, but I absolutely refuse to believe the solution must
>> be as complicated as this. Recall that the baseline is a 30-lines
>> script. I can't bring myself to believe that a four-modules, over
>> thousand lines solution justifies the added complexity.
>
> I count 2 modules and about 800 loc. 2 to 300 of which implements
> functionality which doesn't exist in std.path but should. The ANTLR crap
> could be replaced by a hundred lines of handwritten code, but the
> grammar already existed and took less time.

Thanks for replying to this. I'd been afraid that I was coming off too 
critical. (I counted the ANTLR files as modules, and I think that's 
fair.) To give you an idea on where I come from, distributing dmdz with 
dmd is also a message to users on how things are getting done in D.

For the problem "Compile a D file and all its dependents, link, and run" 
the solution rdmd has 469 lines. It seems quite much to me, but I 
couldn't find ways to make it much smaller.

For the problem "Given a zip file containing a D program, build it" the 
dmdz solution is quite large. If we count everything:

$ wc --lines dmdz.d import/antlrrt/*.d lexd.g opts.d sed.sh
    782 dmdz.d
    891 import/antlrrt/collections.d
    551 import/antlrrt/exceptions.d
   1253 import/antlrrt/lexing.d
   2085 import/antlrrt/parsing.d
     10 import/antlrrt/runtime.d
    600 import/antlrrt/utils.d
    436 lexd.g
     88 opts.d
     13 sed.sh
   6709 total

Arguably we can discount the import stuff, although I'd already raise 
some objections:

$ wc --lines dmdz.d lexd.g opts.d sed.sh
   782 dmdz.d
   436 lexd.g
    88 opts.d
    13 sed.sh
  1319 total

That would suggest that it's about three times as difficult to build 
stuff present in a zip file than to deduce dependencies and build stuff 
not in a zip file. I find that difficult to swallow because to me 
building stuff in a zip file should  be in some ways easier because 
there are no dependencies to deduce - they can be assumed to be in the 
zip file.

I looked more through the program and it looks like it uses the zip 
library (honestly I would have used system("unzip...")), which does add 
some aggravation for arguably a good reason. (But I also see there's no 
caching, which is an important requirement.)

In my mind it was all about check cache, unzip, and build. True there 
are details such as lib vs. executable that can be messy but I don't 
think anything could blow complexity up too hard.

>> Besides, what happened to std.getopt? You don't need to recognize dmd's
>> options any more than rdmd does. rdmd dedicates only a few lines to
>> argument parsing, dmdz makes it a science.
>
> It started when I said, "huh. when is this thing building an executable,
> and when is it building a library?", and parsing dmd's options seemed
> like the most generally useful way of finding that out. I rather like
> the way it's turned out. eg during development:
>
> $ dmdz dxl.zip -unittest
>  > ...
> $ ./dxl/bin/dxl
>  > ...
>
> "alright, unittests pass"
>
> $ dmdz dxl.zip
>  > ...
>
> "now for the release executable"

Nice, but I don't know why you need to understand dmd's flags instead of 
simply forwarding them to dmd. You could define dmdz-specific flags 
which you parse and understand, and then dump everything else to dmd, 
which will figure its own checking and error messages and all that.

> fwiw, I've never used rdmd due to bug 3860.

I didn't mean you to use it as much as look through it for examples of 
patterns that may be useful to dmdz (such as the one above).

>> Don't take this the wrong way, the work is absolutely a tour de force.
>> I'm just saying that things could be dramatically simpler with just a
>> little loss of features. I'm looking over the code and am puzzled about
>> the kind of gunpower that seems to be necessary for achieving the task.
>
> Huh. When all you have is a harquebus ..

Hehe :o). Well definitely you need to submit your stdlib additions to 
e.g. bugzilla.

>> Recall what's needed: someone who is able and willing would like to
>> distribute a multi-module solution as a zip file. dmdz must provide a
>> means to do so. Simple as that. The "able and willing" part is important
>> - you don't need to cope with arbitrarily-formatted archives, you can
>> impose people how the zip must be formatted. If you ask for them to
>> provide a file called "main.d" in the root of the zip, then so be it if
>> it reduces the size of dmdz by a factor of ten.
>>
>>
>> Andrei
>
> By restricting the format of the zip file a bit and moving the directory
> dmd gets run in, I might save 100 loc. Maybe.
>
> Does adding main.d to root help with the run flag? It doesn't do
> anything for dmdz that I can see.
>
> By introducing path2list et al into std.path or wherever (really, it is
> quite handy) and fixing basename and dirname, I could save 2 - 300 loc.
>
> By removing piecemeal and getting rid of dmd flags, I could quit 2 - 300
> loc plus the ANTLR modules. Except I find both of those features
> occasionally useful. Given the choice, I'd keep them.

I think it would be great to remove all stuff that's not necessary. I 
paste at the end of this message my two baselines: a shell script and a 
D program. They compare poorly with your program, but are extremely 
simple. I think it may be useful to see how much impact each feature 
that these programs lack is adding size to your solution.


Andrei


#!/usr/bin/zsh

# Accepted extensions
EXTENSIONS=(d di a o)
# The one and only parameter is the zip file
ZIP=$1
# Target directory
TGT=/tmp/$ZIP
# Binary result is the name of the zip without the .zip
BIN=${ZIP/.zip/}

# Is the zip file in there?
if [[ ! -f $ZIP ]]; then
     echo "Zip file missing: \`$ZIP'" >&2
     echo "Usage: dmdz file.zip" >&2
     exit 1
fi

# Was the zip file already extracted? If not, extract it
if [[ ! -d $TGT ]] || [[ $ZIP -nt $TGT ]]; then
     mkdir --parents $TGT
     unzip $ZIP -d $TGT >/dev/null
fi

# Compile all files with accepted extensions
FIND="find . -type f -false "
for EXT in $EXTENSIONS; do
     FIND="$FIND -or -iname '*.$EXT'"
done
(cd $TGT && dmd -of$BIN `eval $FIND`)


#!/usr/bin/env rdmd

// Accepted extensions
auto extensions = [ "d", "di", "a", "o" ];

int main(string[] args) {
     // The one and only parameter is the zip file
     auto zip = args[1];
     if (!exists(zip)) {
         stderr.writeln("Zip file missing: `", zip, "'");
         stderr.writeln("Usage: dmdz file.zip");
         return 1;
     }
     // Target directory
     auto tgt = "/tmp/" ~ zip;
     // Binary result is the name of the zip without the .zip
     auto bin = replace(zip, ".zip", "");

     // Was the zip file already extracted? If not, extract it
     if (lastModified(zip) >= lastModified(tgt, d_time.min)) {
         system("mkdir --parents " ~ tgt);
         system("unzip " ~ zip " -d " tgt ~ " >/dev/null");
     }

     // Compile all files with accepted extensions
     auto find = "find . -type f -false ";
     foreach (ext; extensions) {
         find ~= " -or -iname '*." ~ ext ~ "'";
     }
     return system("cd " ~ tgt ~ " && dmd -of" ~ bin ~ " `eval " ~ find 
~ "`");
}