[Issue 18225] New: Wrong condition in VRP

[d-bugmail at puremagic.com]

https://issues.dlang.org/show_bug.cgi?id=18225

          Issue ID: 18225
           Summary: Wrong condition in VRP
           Product: D
           Version: D2
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P1
         Component: dmd
          Assignee: nobody at puremagic.com
          Reporter: lucia.mcojocaru at gmail.com

While trying to add coverage to src/dmd/intrange.d, I found an incorrect
condition.
https://github.com/dlang/dmd/blob/master/src/dmd/intrange.d#L555

It's the same as the if condition right above, when it should actually test
that the min and max values of the range have different signs:
(imin.negative ^ imax.negative) == 1

Fixing this condition, I unearthed some issues with the way the sign was
handled by the 2 algorithms in maxAnd and maxOr from intRange.d.

Also ~SignExtendedNumber could lead to -0 (0 with the negative boolean set)
which crashed various tests.

Here are the new coverage tests which fail after the condition is fixed:
void bitOrTest()
{
    ushort a, b;
    byte res = ((a % 127) - 126) | ((b % 6) - 5);
}

void bitAndTest()
{
   ushort a, b;
   byte res = ((a % 7) - 6) & ((b % 7) - 6);
}

void modulus()
{
   short a;
   byte foo = (a - short.max - 1) % 127;
}

These issues appeared after https://github.com/dlang/dmd/pull/7355

Fixes in https://github.com/dlang/dmd/pull/7556

--