Unit tests, asserts and contradictions in the spec

[John Colvin]

On Thursday, 7 February 2019 at 18:06:24 UTC, H. S. Teoh wrote:
> On Thu, Feb 07, 2019 at 04:49:38PM +0000, John Colvin via 
> Digitalmars-d wrote: [...]
>> A fork-based unittest runner would solve some problems without 
>> having to restart the process (could be expensive startup) or 
>> have people re-write their tests to use a new type of assert.
>> 
>> The process is started, static constructors are run setting up 
>> anything needed, the process is then forked & the tests run in 
>> the fork until death from success or assert, somehow 
>> communicating the index of the last successful test to the 
>> runner (let's call that tests[i]). Then if i < test.length - 1 
>> do another fork and start from tests[i + 2] to skip the one 
>> that failed.
>> 
>> There are probably corner cases where you wouldn't want this 
>> behavior, but I can't think of one off the top of my head.
>
> One case is where the unittests depend on the state of the 
> filesystem, e.g., they all write to the same temp file as part 
> of the testing process. I don't recommend this practice, 
> though, for obvious reasons.

Why would this cause a problem? Unless the tests are dependent on 
the *order* they're run it, which is of course madness. (Note 
that I am not suggesting running in parallel and that file 
descriptors would be inherited in the child fork)

Can you sketch out a concrete case?

> (In fact, I'm inclined to say unittests should *not* depend on 
> the filesystem at all; IMNSHO `import std.stdio;` is a code 
> smell in unittests. The code should be refactored such that it 
> is testable without involving the actual filesystem (simulated 
> filesystems can be used for testing FS-dependent logic, if the 
> API is designed properly). But I bet most D unittests aren't 
> written up to that standard!)
>
>
> T

I reckon most are reasonably good w.r.t. this, by virtue of being 
simple unittests of pure-ish code.