Speeding up importing Phobos files

[Neia Neutuladh]

On Sat, 19 Jan 2019 00:45:27 -0800, Walter Bright wrote:
> Andrei and I were talking on the phone today, trading ideas about
> speeding up importation of Phobos files. Any particular D file tends to
> import much of Phobos, and much of Phobos imports the rest of it. We've
> both noticed that file size doesn't seem to matter much for importation
> speed, but file lookups remain slow.
> 
> So looking up fewer files would make it faster.

I compiled one file with one extra -I directive as a test. The file had 
two imports, one for url.d (which depends on std.conv and std.string) and 
one for std.stdio.

For each transitive import encountered, the compiler:
* looked at each import path (total of four: ., druntime, phobos, and 
urld's source dir)
* looked at each possible extension (.d, .di, and none)
* built a matching filename
* checked if it existed

Maybe it should do a shallow directory listing for the current directory 
and every -I path at startup, using that information to prune the list of 
checks it needs to do. It could also do that recursively when importing 
from a subpackage. Then it would have called 'opendir' about four times 
and readdir() about 70 times; it wouldn't ever have had to call exists(). 
It would allocate a lot less memory. And that's a change that will help 
people who don't zip up their source code every time they want to compile 
it.

It could even do this in another thread while parsing the files passed on 
the command line. That would require a bit of caution, of course.