Parallel execution of unittests

[Bruno Medeiros via Digitalmars-d]

On 06/05/2014 18:58, Dicebot wrote:
> On Tuesday, 6 May 2014 at 15:54:30 UTC, Bruno Medeiros wrote:
>> But before we continue the discussion, we are missing am more basic
>> assumption here: Do we want D to have a Unit-Testing facility, or a
>> Testing facility?? In other words, do we want to be able to write
>> automated tests that are Integration tests or just Unit tests? Because
>> if we go with this option of making D unittest blocks run in parallel,
>> we kill the option of them supporting Integration Tests. I don't think
>> this is good.
>
> These days I often find myself leaning towards writing mostly
> integration tests with only limited amount of unit tests. But writing
> good integration test is very different from writing good unit test and
> usually implies quite lot of boilerplate. Truth is D does not currently
> have any higher-level facility at all. It has an _awesome_ unit test
> facility which gets often misused for writing sloppy integration tests.
>

Indeed: I also find myself writing more integration tests than unit 
tests, at least in the way I consider an integration tests to be (in 
some cases the distinction between integration and unit test may not be 
very easy or clear, IMO).

> I'd love to keep existing facility as-is and think about providing good
> library augmentation for any sort of higher level approach.
>

The unittest block is enough right now to support integration tests. To 
support common test fixture setup (akin to @Before and @After in xUnit), 
perhaps some syntax sugar could be added, although with D's language 
facilities (meta-programming, functional constructs, scope statements, 
etc.), we can do pretty well with existing functionality already.

>
> Good integration test is very different. It has certain assumptions
> about initial system state and notifies user if those are not met. It
> can take ages to run and can test real-world situations. It is not
> supposed to be run implicitly and frequently. You don't want to keep
> your integration tests inline because of amount of boilerplate code
> those usually need.
>

They are somewhat different. I wouldn't say very different. I don't 
agree integration tests usually take ages to run. Some of them can run 
fairly fast too, and are executed as frequently as unit tests.
In DDT for example I always run unit tests the same time as integration 
tests. As I said, I don't find it useful to have a strict distinction 
between those.
Rather, if I want to run a subset of tests, usually what I filter on is 
running only the tests of a certain plugin (DDT has 3 plugins), Java 
package, or Java class.
Additionally the parser tests can be run in what I call "Lite" mode, 
which instead of running the full test suite, skips some of the 
heavyweight, parameterized tests, to make the suite run faster. Most of 
the cases are generated from templates, others are blind mass parse 
tests, such as parsing all source modules in Phobos. But what "Lite" 
mode cuts is not Integration tests, but rather the input sets of 
parameterized tests.

As for keeping integration tests inline or not, yeah, you are likely to 
prefer putting them in a separate file. Doesn't mean we need a different 
language construct other than the unittest block for that.


> I see no good in trying to unite those very different beasts and my
> experience with existing test libraries has been very unpleasant in that
> regard.

What test libraries/frameworks have you used?

-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros