DustMite, a D test case minimization tool

[Don]

Vladimir Panteleev wrote:
> Inspired by Tigris Delta and the "Want to help DMD bugfixing? Write a 
> simple utility" thread from digitalmars.D.learn. I hope the DMD 
> development team will find this useful.
> 
> Advantages over Tigris delta:
> 
> * Easy to use (takes only two arguments, no need to fiddle with levels)
> * Readable output (comments and indentation are preserved)
> * Native support for multiple files (accepts a path to an entire 
> directory for input)
> * Written for D
> * Written in D
> * Not written in Perl
> * Can recognize constructs such as try/catch, function invariants 
> (in/out/body)
> * Only 440 lines of source code
> 
> If you've never used delta: this is a tool which attempts to shrink 
> files by deleting fragments iteratively, as long as the file satisfies a 
> user-specified condition (for example, a specific error message when 
> passed through the compiler).
> 
> Usage:
> 
> 1. Formulate a condition command, which should exit with a status code 
> of 0 when DustMite is on the right track, and anything else otherwise.
>    Example: dmd test.d 2>&1 | grep -qF "Assertion failed"
> 2. Place all the files that dustmite is to minimize in a new directory.
> 3. If you'd like to test your condition command, don't forget to clean 
> up temporary files afterwards.
> 4. Run: dustmite path/to/directory test-command
> 5. After a while, dustmite will finish working and create 
> path/to/directory.reduced
> 
> I've tested it with a self-induced "bug" in std.datetime, it seems to 
> work great. If you find that it breaks on something, let me know.
> 
> https://github.com/CyberShadow/DustMite
> 

This is fantastic for ICE bugs. But it doesn't work well for certain 
types of bugs, such as the one I tracked down recently, which are of the 
form:
assert(foo()==30);
int foo()
{
    if (func1())
      return 0;
    return func2();
}
It reduces foo() to: int foo() { return 0; }
which causes the test to still fail, but the bug is gone.
The --noremove option doesn't help much because you have to name all of 
the functions which have that behaviour, and you don't know them in advance.

I was pretty surprised to find that reduced test cases still had 
comments in them. They should be the first things to be stripped out.

----

Something which has occured to me is that nearly always when I'm 
reducing a bug, the code which is actually getting run is close to 
minimal. So running with -cov and then removing any line with 0000 hits
would probably be pretty effective. Of course, you need to make sure it 
still compiles, which could be quite difficult. You could run -cov in 
the first run, and then when you slice up the file, you could mark 
sections as 'hit' (count>0), 'missed'(count==0), or 'void' (no count, 
could be a declaration, or a comment).
Generally any section with a 'hit' shouldn't be removed, but the test 
case isn't minimal until all 'missed' lines are gone. Most of the 'void' 
lines are probably removable too (it just won't compile, if they are 
necessary).