Profile-Guided Optimization (PGO) support in D ecosystem

[Siarhei Siamashka]

On Sunday, 12 November 2023 at 18:03:19 UTC, max haughton wrote:
> If you search for PGO in the dmd repo you will find that I 
> implemented a pgo build for the compiler a while ago.

The PGO code is there, but it seems to fail at compiling 
"dshell_prebuilt.d" file. And aborts collecting the profiling 
data prematurely because of this:

```
[...]
Built dmd with PGO instrumentation
Compiling dmd testsuite to generate PGO data
Executing: ldmd2 -m64 
-of/tmp/dmd/compiler/test/test_results/d_do_test 
/tmp/dmd/compiler/test/tools/d_do_test.d -fPIC 
-I/tmp/dmd/compiler/test/tools -i -version=NoMain
Executing: ldmd2 -m64 
-of/tmp/dmd/compiler/test/test_results/unit_test_runner 
/tmp/dmd/compiler/test/tools/unit_test_runner.d -fPIC 
/tmp/dmd/compiler/test/tools/paths
Executing: /tmp/dmd/generated/linux/release/64/dmd -conf= -m64 
-of/tmp/dmd/compiler/test/test_results/dshell_prebuilt.o -c 
/tmp/dmd/compiler/test/tools/dshell_prebuilt/dshell_prebuilt.d 
-fPIC
Executing: ldmd2 -m64 
-of/tmp/dmd/compiler/test/test_results/sanitize_json 
/tmp/dmd/compiler/test/tools/sanitize_json.d -fPIC
/tmp/dmd/compiler/test/tools/dshell_prebuilt/dshell_prebuilt.d(6): Error: unable to read module `stdlib`
/tmp/dmd/compiler/test/tools/dshell_prebuilt/dshell_prebuilt.d(6):        Expected 'core/stdc/stdlib.d' or 'core/stdc/stdlib/package.d' in one of the following import paths:
import path[0] = /tmp/dmd/compiler/test/../../druntime/import
import path[1] = /tmp/dmd/compiler/test/../../../phobos
failed to build 
'/tmp/dmd/compiler/test/test_results/dshell_prebuilt.o'
dmd tests failed! This will not end the PGO build because some 
data may have been gathered
Merging PGO data
[...]
```

The compiler is still built successfully, but its performance is 
not optimal this way. I patched up "dshell_prebuilt.d" to strip 
everything out of it, the error disappears, the whole DMD test 
suite seems to compile and the compiler gets a noticeable 
performance boost.

> I'm not sure if it's enabled for releases

This doesn't seem to be the case at least for the 
https://downloads.dlang.org/releases/2.x/2.106.0/dmd.2.106.0.linux.tar.xz tarball. LTO is enabled, but apparently without PGO.

> but we use it internally at Symmetry IIRC.

It's good to know that DMD with LTO+PGO is already successfully 
used in production at Symmetry. Would it make sense to also 
enable this optimization for everyone else?