Profile-Guided Optimization (PGO) support in D ecosystem

[Siarhei Siamashka]

On Monday, 18 December 2023 at 02:30:21 UTC, max haughton wrote:
> On Monday, 18 December 2023 at 01:57:17 UTC, Siarhei Siamashka 
> wrote:
>> On Sunday, 12 November 2023 at 18:03:19 UTC, max haughton 
>> wrote:
>>> [...]
>>
>> The PGO code is there, but it seems to fail at compiling 
>> "dshell_prebuilt.d" file. And aborts collecting the profiling 
>> data prematurely because of this:
>>
>> [...]
>
> Does druntime need building?

That's a good question. As the author of this code, you probably 
have a much better idea about how it's supposed to work. I tried 
to come up with some scriptable step by step build instructions:

```
DMD_TAG=v2.106.0
LDMD=ldmd2-1.32.0

git clone --depth 1 --branch "${DMD_TAG}" 
https://github.com/dlang/dmd.git || exit 1
git clone --depth 1 --branch "${DMD_TAG}" 
https://github.com/dlang/phobos.git || exit 1

cd dmd

make -j4 -f posix.mak HOST_DMD=$LDMD ENABLE_RELEASE=1 
ENABLE_LTO=1 || exit 1

cd ../phobos

make -j4 -f posix.mak || exit 1

cd ../dmd

cp generated/linux/release/64/dmd "../dmd_${DMD_TAG}_lto"

rm -rf generated

rdmd compiler/src/build.d OS="linux" BUILD="release" MODEL="64" 
HOST_DMD="$LDMD" CXX="c++" AUTO_BOOTSTRAP="" DOCDIR="" STDDOC="" 
DOC_OUTPUT_DIR="" MAKE="make" VERBOSE="" ENABLE_RELEASE="1" 
ENABLE_DEBUG="" ENABLE_ASSERTS="" ENABLE_LTO="1" 
ENABLE_UNITTEST="" ENABLE_PROFILE="" ENABLE_COVERAGE="" DFLAGS="" 
dmd-pgo || exit 1

cp generated/linux/release/64/dmd "../dmd_${DMD_TAG}_lto+pgo"

ls -l dmd_*
```

Running it results in the following:

```
-rwxr-xr-x 1 ssvb ssvb 7626560 Dec 18 13:30 dmd_v2.106.0_lto
-rwxr-xr-x 1 ssvb ssvb 7994232 Dec 18 13:38 dmd_v2.106.0_lto+pgo
```

The `dmd_v2.106.0_lto` file roughly matches the size and 
performance characteristics of the `dmd` executable from the 
https://downloads.dlang.org/releases/2.x/2.106.0/dmd.2.106.0.linux.tar.xz release tarball and `dmd_v2.106.0_lto+pgo` is its faster PGO-enabled upgrade. I can observe at least 10% compilation time reduction when using the PGO-enabled `dmd`.

It's rather messy, but this somehow works. There are many 
questions though. For example, should the "dmd-pgo" target be 
accessible from the makefile without invoking "rdmd 
compiler/src/build.d" directly? Is sharing the same directory 
"generated/linux/release" for the produced non-PGO and PGO 
binaries actually okay? Is the dmd testsuite a good training set 
or maybe collecting profiling data during Phobos compilation 
would be better? The "dshell_prebuilt.d" glitch happens if the 
PGO-enabled DMD is built before Phobos & druntime and this makes 
everything fragile and non-intuitive. So the LTO-enabled DMD 
needs to be built first, then we need to use it to compile 
druntime, and finally the "generated/linux/release" directory has 
to be erased before the PGO build is started in order not to 
clash with it.

Either way, providing faster PGO-enabled binary releases of DMD 
would make it more competitive in the compilation speed race 
against LDC: 
https://forum.dlang.org/post/pugqkvthbicqaigemijj@forum.dlang.org 
:-)