Problem with GC and address/leak sanitizer

[Luís Marques]

On Friday, 7 March 2025 at 14:51:22 UTC, Luís Marques wrote:
> LDC 1.40.0 (git commit 9296fd6fcc) reproduces the problem with 
> both LLVM 15.0 and 19.1, so for now it seems that this isn't 
> related to something having changed on the LLVM side.

I finally found some time today to analyze this properly, rather 
than just trying different LDC/LLVM versions. I haven't yet 
double-checked everything, but based on my current understanding, 
I believe I've identified the issue, along with several 
deficiencies:

1. **The main issue:** The distributed 
`libdruntime-ldc-shared.so` appears to be compiled without 
`version=SupportSanitizers`, so the fake stack scanning isn't 
enabled. I conclude this by observing that the distributed 
runtime's disassembly doesn't include 
`scanStackForASanFakeStack`, unlike when this option is enabled. 
The necessary code exists in the runtime's source, but it's 
simply not being compiled in.

2. **Fake stack support should work when properly enabled:** If 
you build LDC with `-DRT_SUPPORT_SANITIZERS=True`, the fake stack 
is scanned as expected. From what I can see (by checking for the 
proper GC scan calls and addresses), everything is being done 
correctly. The test `tests/sanitizers/asan_fakestack_GC.d` still 
fails, but not because of broken fake stack scanning—instead, the 
test itself is broken. An assertion fails in 
`test_non_null_does_not_trigger_collection` because memory from a 
previous iteration is being freed, not because it's freeing 
memory incorrectly for the current iteration.

3. **Redundant function calls:** The runtime function 
`scanAllTypeImpl` calls `scanStackForASanFakeStack` multiple 
times for the same fake stack frame, using identical memory 
ranges each time. While technically correct, this is unnecessary 
and inefficient. Based on the comment `"This is the pointer we 
are catching with"`, it seems there's a misunderstanding 
regarding how the fake stack should be located.

4. **Misleading comment in the test:** The comment `// Large 
enough so it will be on fakestack heap (not inlined in local 
stack frame)` in `asan_fakestack_GC.d` appears to be incorrect. 
The size of the allocation shouldn't determine this—if its 
address escapes, it must go into the heap-allocated fake stack 
regardless.