DustMite, a D test case minimization tool

[Don]

Don wrote:
> Vladimir Panteleev wrote:
>> Inspired by Tigris Delta and the "Want to help DMD bugfixing? Write a 
>> simple utility" thread from digitalmars.D.learn. I hope the DMD 
>> development team will find this useful.
>>
>> Advantages over Tigris delta:
>>
>> * Easy to use (takes only two arguments, no need to fiddle with levels)
>> * Readable output (comments and indentation are preserved)
>> * Native support for multiple files (accepts a path to an entire 
>> directory for input)
>> * Written for D
>> * Written in D
>> * Not written in Perl
>> * Can recognize constructs such as try/catch, function invariants 
>> (in/out/body)
>> * Only 440 lines of source code
>>
>> If you've never used delta: this is a tool which attempts to shrink 
>> files by deleting fragments iteratively, as long as the file satisfies 
>> a user-specified condition (for example, a specific error message when 
>> passed through the compiler).
>>
>> Usage:
>>
>> 1. Formulate a condition command, which should exit with a status code 
>> of 0 when DustMite is on the right track, and anything else otherwise.
>>    Example: dmd test.d 2>&1 | grep -qF "Assertion failed"
>> 2. Place all the files that dustmite is to minimize in a new directory.
>> 3. If you'd like to test your condition command, don't forget to clean 
>> up temporary files afterwards.
>> 4. Run: dustmite path/to/directory test-command
>> 5. After a while, dustmite will finish working and create 
>> path/to/directory.reduced
>>
>> I've tested it with a self-induced "bug" in std.datetime, it seems to 
>> work great. If you find that it breaks on something, let me know.
>>
>> https://github.com/CyberShadow/DustMite
>>
> 
> This is fantastic for ICE bugs. But it doesn't work well for certain 
> types of bugs, such as the one I tracked down recently, which are of the 
> form:
> assert(foo()==30);
> int foo()
> {
>    if (func1())
>      return 0;
>    return func2();
> }
> It reduces foo() to: int foo() { return 0; }
> which causes the test to still fail, but the bug is gone.
> The --noremove option doesn't help much because you have to name all of 
> the functions which have that behaviour, and you don't know them in 
> advance.
> 
> I was pretty surprised to find that reduced test cases still had 
> comments in them. They should be the first things to be stripped out.
> 
> ----
> 
> Something which has occured to me is that nearly always when I'm 
> reducing a bug, the code which is actually getting run is close to 
> minimal. So running with -cov and then removing any line with 0000 hits
> would probably be pretty effective. Of course, you need to make sure it 
> still compiles, which could be quite difficult. You could run -cov in 
> the first run, and then when you slice up the file, you could mark 
> sections as 'hit' (count>0), 'missed'(count==0), or 'void' (no count, 
> could be a declaration, or a comment).
> Generally any section with a 'hit' shouldn't be removed, but the test 
> case isn't minimal until all 'missed' lines are gone. Most of the 'void' 
> lines are probably removable too (it just won't compile, if they are 
> necessary).

Specific suggestion:
When dustmite does the initial slicing of the file, it should look for a 
matching .lst file. If it exists, then for any line in the .lst file 
where there is a '1'..'9' before the first non-numeric character on that 
line, that corresponding line in the .d file should be ineligible for 
deletion. (This is recursive; the function it's in shouldn't be 
deletable in its entirity).
Everything else should be unchanged.
Then, for these unstable bugs, the user just needs to compile with -cov 
first, and retain the .lst files in the directory.