Speeding up importing Phobos files
Amex at gmail.com
Sat Jun 8 06:29:16 UTC 2019
On Saturday, 19 January 2019 at 08:45:27 UTC, Walter Bright wrote:
> Andrei and I were talking on the phone today, trading ideas
> about speeding up importation of Phobos files. Any particular D
> file tends to import much of Phobos, and much of Phobos imports
> the rest of it. We've both noticed that file size doesn't seem
> to matter much for importation speed, but file lookups remain
> So looking up fewer files would make it faster.
> Here's the idea: Place all Phobos source files into a single
> zip file, call it phobos.zip (duh). Then,
> dmd myfile.d phobos.zip
> and the compiler will look in phobos.zip to resolve, say,
> std/stdio.d. If phobos.zip is opened as a memory mapped file,
> whenever std/stdio.d is read, the file will be "faulted" into
> memory rather than doing a file lookup / read. We're
> speculating that this should be significantly faster, besides
> being very convenient for the user to treat Phobos as a single
> file rather than a blizzard. (phobos.lib could also be in the
> same file!)
> It doesn't have to be just phobos, this can be a general
> facility. People can distribute their D libraries as a zip file
> that never needs unzipping.
> We already have https://dlang.org/phobos/std_zip.html to do the
> dirty work. We can experiment to see if compressed zips are
> faster than uncompressed ones.
> This can be a fun challenge! Anyone up for it?
> P.S. dmd's ability to directly manipulate object library files,
> rather than going through lib or ar, has been a nice success.
Why not compile phobos to an object file? Basically store the AST
directly in to a file and just load it. Phobos never changes so
why recompile it over and over and over and over?
This should be done with all files... sorta like rdmd, so to
It might take some work to figure out how to get it all to work
but maybe the time has come to stop using ancient design patterns
and move on to a higher level?
After all, the issue is mainly templates since they cannot be
compiled to a library... but if they could, then they wouldn't be
an issue. .til -> .lib -> .exe
.til is a higher level library that includes objectified
templates from D code, it is basically an extension of a lib file
and eventually the lib is compiled in to the exe.
More information about the Digitalmars-d