[Issue 9242] New: Add stack smashing code to flush out heisenbugs

[d-bugmail at puremagic.com]

http://d.puremagic.com/issues/show_bug.cgi?id=9242

           Summary: Add stack smashing code to flush out heisenbugs
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody at puremagic.com
        ReportedBy: bugzilla at digitalmars.com


--- Comment #0 from Walter Bright <bugzilla at digitalmars.com> 2012-12-29 16:42:28 PST ---
We've lately had some very hard to track down heisenbugs that ultimately turned
out to be references to stack frames that have gone out of scope. This
particularly is happening when there are bugs in the lambda implementation, but
it is quite possible that such can still happen with user code.

It's not possible to always detect these at runtime, but their incidence can be
reduced, and bugs should be easier to track down because those references will
not randomly appear to work.

The first part is to replace the stack frame cleanup code:

    mov ESP,EBP
    pop EBP
    ret

with:

    call __stack_frame_smash
    mov ESP,EBP
    pop EBP
    ret

What __stack_frame_smash does is:

    1. set all memory [ESP..EBP] to something like 0xDEADBEEF
    2. set to 0xDEADBEEF all registers that are not guaranteed to be preserved
    across function calls.

Unfortunately, this won't smash the parameter stack, and it can't because the
callee cannot know how many parameters are on that stack (according to the
ABI). But, ya can't have everything.

The second part is, when a pointer, reference, dynamic array, or delegate is
returned from a function, add the following code to the epilog before the call
to __stack_frame_smash:

    cmp EAX,EBP
    ja  Ok
    cmp EAX,ESP
    jb  Ok
    halt
Ok:

or EDX in the case of dynamic arrays. This will halt the machine if a pointer
into the deallocated stack frame is returned.

Insertion of this code is done if the -gh switch is thrown.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------