[Issue 16274] New: The curses of debugging: short argument passed in 16-bit register, against ABI

[via Digitalmars-d-bugs]

https://issues.dlang.org/show_bug.cgi?id=16274

          Issue ID: 16274
           Summary: The curses of debugging: short argument passed in
                    16-bit register, against ABI
           Product: D
           Version: D2
          Hardware: x86
                OS: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P1
         Component: dmd
          Assignee: nobody at puremagic.com
          Reporter: luis at luismarques.eu

On my system (OS X 10.11.5 (15F34), 64-bit), if you compile (with DMD) and run
the following program you'll get an executable that behaves incorrectly about
half of the executions:

    extern(C)
    {
        void* initscr();
        int start_color();
        int init_pair(short pair, short f, short b);
        int attron(int attrs);
        int mvprintw(int y, int x, const(char)* fmt, ...);
        int attroff(int attrs);
        int getch();
        int endwin();
        int refresh();
    }

    enum
    {
        COLOR_BLACK   = 0,
        COLOR_RED     = 1,
        COLOR_GREEN   = 2,
        COLOR_YELLOW  = 3,
        COLOR_BLUE    = 4,
        COLOR_MAGENTA = 5,
        COLOR_CYAN    = 6,
        COLOR_WHITE   = 7
    }

    int COLOR_PAIR(int n)
    {
        return n << 8;
    }

    void main()
    {
        initscr();
        start_color();
        init_pair(1, COLOR_WHITE, COLOR_BLUE);
        int pair = COLOR_PAIR(1);
        attron(pair);
        mvprintw(1, 1, "Test");
        attroff(pair);
        refresh();
        getch();
        endwin();
    }

    $ rdmd -L-lcurses test.d

If all is well, you'll see the text "Test" in white text on a blue background.
If the bug is exercised then you won't see any text. Sometimes it takes quite a
few number of executions until it starts or stops exhibiting the problem.

This reduced text case was taken very painstakingly from a large codebase,
where changing unrelated things around would make the problem come and go ---
probably; it was sometimes hard to tell, given the non-deterministic bug.

Currently, it seems that this program always works fine when using LDC,
although my vague recollection is that this wasn't true when I had last looked
at this issue (which was before the latest LDC). That might be because of the
following difference in compilation:

LDC:
        __Dmain:
        subq    $0x38, %rsp
        callq   _initscr
        movq    %rax, 0x28(%rsp)
        callq   _start_color
        movl    $0x1, %edi   <---
        movl    $0x7, %esi   <---
        movl    $0x4, %edx  <---
        movl    %eax, 0x24(%rsp)
        callq   _init_pair

DMD:
        __Dmain:
        pushq   %rbp
        movq    %rsp, %rbp
        subq    $0x10, %rsp
        callq   _initscr
        callq   _start_color
        movw    $0x4, %dx  <---
        movw    $0x7, %si    <---
        movw    $0x1, %di   <---
        callq   _init_pair

And indeed, if you change the code to the following the problem goes away:

    void main()
    {
        initscr();
        start_color();

        asm
        {
            xor EDX, EDX;
            xor ESI, ESI;
            xor EDI, EDI;
        }

        init_pair(1, COLOR_WHITE, COLOR_BLUE);
        int pair = COLOR_PAIR(1);
        attron(pair);
        mvprintw(1, 1, "Test");
        attroff(pair);
        refresh();
        getch();
        endwin();
    }

I guess the ABI requires the whole register to be used for a 16-bit value?

The docs for my system say "The OS X x86-64 function calling conventions are
the same as the function calling conventions described in System V Application
Binary Interface AMD64 Architecture Processor Supplement, found at
http://people.freebsd.org/~obrien/amd64-elf-abi.pdf. See that document for
details."

In that document I found the following:

    • Arguments of types (signed and unsigned) _Bool, char, short, int,
long, long long, and pointers are in the INTEGER class

     (...)

    2. If the class is INTEGER, the next available register of the sequence
%rdi,
%rsi, %rdx, %rcx, %r8 and %r9 is used

So I guess this is a DMD code generation bug?

--