Building the compiler in 2 seconds with `dmd -i`

[H. S. Teoh]

On Fri, May 19, 2023 at 09:13:02AM -0700, Ali Çehreli via Digitalmars-d wrote:
> On 5/19/23 07:28, Adam D Ruppe wrote:
> 
> > dmd -i is heaven. I, without a doubt, place it as the most
> > significant addition to the D ecosystem of the last six years
> > (probably more if I extended the audit)
> >
> > I've deleted so many convoluted builds in favor of it and always
> > seen wins.
> 
> -i is great and I've started with -i in my last project that uses
> Makefiles.  For now, I went back to manual dependencies by listing all
> the modules explicitly. Otherwise, 'make' did not know modifications
> to the -i files. I think I will incorporate the -deps switch later.
[...]

I propose a new compiler switch that does the following:
- For every file included by -i, check whether it has changed since the
  last compile.
- If none of the input files, including those pulled in by -i, have
  changed since the last compile, skip all semantic and codegen steps
  and exit with a success status.

Then you could just unconditionally run {dmd,ldc2} -i every time, and it
will just be a (mostly) no-op if none of the source files have changed.

//

The big question is, of course, how will the compiler know whether a
file has changed?  There are several ways of doing this.

- The age-old Make way of checking timestamps. OK if you're on the
  filesystem, but unreliable and prone to false positives and false
  negatives. False positives are not horrible -- the worst is you just
  recompile something that didn't change, no problem, you get the same
  binary. But false negatives are very bad: the compiler gives you the
  impression that it has recompiled the binary but actually did nothing.
  But in the general case this shouldn't happen (not frequently anyway),
  so a workaround could just be to recompile without the new switch when
  you suspect false negatives are happening.

- Do a quick checksum of each imported file and compare against some
  stored checksums. This could be stored in the executable in an
  optional section that gets ignored by the linker, for example. The
  compiler loads this section and compares the checksum of each source
  file against it. If the checksum is the same, the source file hasn't
  changed. If the checksum is different, or if there's no checksum
  loaded (this is the first compilation of that file), compile as usual.
  The advantage of this approach is that if your checksum algorithm is
  reliable, false positives and false negatives should (almost) never
  happen. The disadvantage is that this will be slow.

//

The above scheme, of course, falls down if you do tricky things like
having string mixins that construct import statements:

	string makeCode(string x, string y) {
		return x ~ " " ~ y ~ ";";
	}
	void main() {
		mixin(makeCode("import", "mymodule");
	}

The compiler wouldn't know that mymodule.d is being pulled in without
running semantic. So the savings will be less, possibly much less, and
you might as well just recompile everything every time instead.


T

-- 
Mediocrity has been pushed to extremes.