Better error message for endless recursion (or other segfaults) on linux

Dennis dkorpel at gmail.com
Sat Jun 17 22:17:18 UTC 2023


When you accidentally put endless recursion in your code, by 
default on linux, you get this printed to the console once the 
stack overflows:

```
[1]    37856 segmentation fault (core dumped)  ./program
```

Not very informative. After this, you typically open a debugger 
like `gdb` and use the `backtrace` command to find where this 
happened. Can't we just print a backtrace when it happens, just 
like an assertion failure?

Well, there exists a `etc.linux.memoryerror` module, which 
provides the `registerMemoryErrorHandler` function. After that is 
called, segfaults are caught using the posix function 
`sigaction(SIGSEGV, ...)`, and it throws a `NullPointerError` or 
`InvalidPointerError` object with a stack trace.

This doesn't work with stack overflow however, because the signal 
callback function uses stack memory itself, so you get a stack 
overflow while handling the stack overflow, and the program just 
aborts again.

This can be solved by using `sigaltstack`, which provides 
alternative stack memory for the signal handler. I tried 
integrating this into the existing code, but it hijacks RIP (the 
instruction pointer) to an assembly routine and continues, so 
custom X86 assembly gets executed in the context of the segfault. 
In the case of stack overflow, this results in a loop where the 
signal handler is called endlessly.

As far as I can see, the assembly tricks are only needed to 
support catching the Error object, which is bad practice any way, 
so I thought it would be simpler to make a new handler using 
`assert(0)`. I also think it can be enabled by default, at the 
very least in debug builds. Here's a PR with my work so far (at 
the time of writing):

https://github.com/dlang/dmd/pull/15331

It seems to works well, but I'm not experienced with signal 
handling, so review comments are welcome. Is it bad for 
performance? Is it unsafe? Can this break existing code? 
Hopefully not!



More information about the Digitalmars-d mailing list