Integration tests

[Russel Winder]

On Fri, 2020-04-17 at 17:51 +0000, Jon Degenhardt via Digitalmars-d-
learn wrote:
> On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:
> > Hi,
> > 
> > Thinking of trying to do the next project in D rather than 
> > Rust, but…
> > 
> > Rust has built in unit testing on a module basis. D has this so 
> > no problem.
> > 
> > Rust allows for integration tests in the tests directory of a 
> > project. These are automatically build and run along with all 
> > unit tests as part of "cargo test".
> > 
> > Does D have any integrated support for integration tests in the 
> > way
> > Rust does?
> 
> Automated testing is important, perhaps you describe further 
> what's needed? I haven't worked with Rust test frameworks, but I 
> took a look at the description of the integration tests and unit 
> tests. It wasn't immediately obvious what can be done with the 
> Rust integration test framework that cannot be done with D's 
> unittest framework.

I should point out that Rust is far behind Python's PyTest in terms of
testing sophistication, and I fear D lags behind Rust. 

I think the following is just saying what everyone knows, but it seems
making it explicit for this thread.

Let us distinguish unit testing, integration testing, and system
testing using the definitions: unit testing is self standing testing
the code with no extra dependencies; integration testing is testing
some functionality of the system with all external resources mocked out
(possibly by using a process to simulate the external resource, which
is what I want for this project); system testing is testing some
functionality with real external resources.

D, like Rust, supports unit testing very nicely, well unit_threading
seems essential, on a per module basis. Python unit testing with PyTest
is very straightforward. 

Rust supports integration testing explicitly, D does not. Python PyTest
does but only because with Python you need no extra infrastructure.
Integration testing must not have access to the internals of a module
or group of modules, but should appear as a using application.
Rust/Cargo integrates this using the tests directory as a peer to the
src directory: each file in the tests directory represents a crate that
uses the code under test as a crate.

Integration and automated system testing in Rust is really a question
of whether external resources are mocked or used for real. Python
PyTest integration and system testing is very much the same.

As far as I know, D/Dub, D/Meson, D/SCons, D/CMake, and D/Make do not
provide an out of the box way of building integration tests.

Hidden agenda item: system testing really needs to be in a sandbox of
some sort. 

> An important concept described was testing a module as an 
> external caller. That would seem very be doable using D's 
> unittest framework. For example, one could create a set of tests 
> against Phobos, put them in a separate location (e.g. a separate 
> file), and arrange to have the unittests run as part of a CI 
> process run along with a build.
> 
> My look was very superficial, perhaps you could explain more.

The important difference between using a unit test infrastructure and
an integration/system test infrastructure is that:

– unit tests are inside the system testing functions and other code
features;
– integration tests are outside the system testing functionality with
mocked external resources.

So the build system needs to build all the codes that realise the 
mocks of the external resources. So for a vestigial Rust project:

.
├── Cargo.lock
├── Cargo.toml
├── README.md
├── src
│   ├── arcam_protocol.rs
│   ├── lib.rs
│   └── main.rs
└── tests
    ├── integration_tests.rs
    └── mock_avr850.rs

src/arcam_protocol/rs has all the unit tests in it. D can do the
equivalent.

tests/mock_avr850.rs is an application that mocks a real amplifier;
tests/integration_tests is an application that uses the project as an
external crate and starts the mock amplifier for each of the tests. As
far as I can tell there is no Dub, Meson, Scons, CMake, Make
infrastructure for building D code that supports this sort of thing.

Python using PyTest can do all the above really very trivially, D/Cargo
has yet to catch up!


-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20200420/d28edc97/attachment-0001.sig>