Thoughts about unittest run order

[Jacob Carlborg]

On 2019-05-06 20:13, H. S. Teoh wrote:
> In theory, the order in which unittests are run ought to be irrelevant.
> In practice, however, the order can either make debugging code changes
> quite easy, or very frustrating.
> 
> I came from a C/C++ background, and so out of pure habit write things
> "backwards", i.e., main() is at the bottom of the file and the stuff
> that main() calls come just before it, and the stuff *they* call come
> before them, etc., and at the top are type declarations and low-level
> functions that later stuff in the module depend on.  After reading one
> of Walter's articles recently about improving the way you write code, I
> decided on a whim to write a helper utility in one of my projects "right
> side up", since D doesn't actually require declarations before usage
> like C/C++ do.  That is, main() goes at the very top, then the stuff
> that main() calls, and so on, with the low-level stuff all the way at
> the bottom of the file.
> 
> It was all going well, until I began to rewrite some of the low-level
> code in the process of adding new features. D's unittests have been
> immensely helpful when I refactor code, since they catch any obvious
> bugs and regressions early on so I don't have to worry too much about
> making large changes.  So I set about rewriting some low-level stuff
> that required extensive changes, relying on the unittests to catch
> mistakes.
> 
> But then I ran into a problem: because D's unittests are currently
> defined to run in lexical order, that means the unittests for
> higher-level functions will run first, followed by the lower-level
> unittests, because of the order I put the code in.  So when I
> accidentally introduced a bug in lower-level code, it was a high-level
> unittest that failed first -- which is too high up to figure out where
> exactly the real bug was. I had to gradually narrow it down from the
> high-level call through the middle-level calls and work my way to the
> low-level function where the bug was introduced.
> 
> This is quite the opposite from my usual experience with "upside-down
> order" code: since the low-level code and unittests would appear first
> in the module, any bugs in the low-level code would trigger failure in
> the low-level unittests first, right where the problem was. Once I fix
> the code to pass those tests, then the higher-level unittests would run
> to ensure the low-level changes didn't break any behaviour the
> higher-level functions were depending on.  This made development faster
> as less time was spent narrowing down why a high-level unittest was
> failing.
> 
> So now I'm tempted to switch back to "upside-down" coding order.
> 
> What do you guys think about this?

There are different schools on how to order code in a file. Some will 
say put all the public functions first and then the private. Some will 
say code should read like a newspaper article, first an overview and the 
more and more you read you get deeper into the details. Others will says 
that you put related code next to each other, regardless if it's public 
or private symbols. I usually put public symbols first and the private 
symbols.

When it comes to the order of unit tests, I think they should run in 
random order. If a test fails it should print a seed value. If the tests 
are run with this seed value the tests should be run in the same order 
as before. This helps with debugging if some tests are accidentally 
depending on each other.

The problem you're facing, I'm guessing, is you run with the default 
unit test runner? If a single test fails, it will stop and run no other 
tests in that module (tests in other modules will run). If you pick and 
existing unit test framework or write your own unit test runner you can 
have the unit tests continue after a failure. Then you would see that 
the lower level tests are failing as well.

Writing a custom unit test runner with the help of the "getUnitTests" 
trait [1] would allow you to do additional things like looking for UDAs 
that could set the order of the unit tests or group the unit tests in 
various ways. You could group them in high and low level groups and have 
the unit test runner run the low level tests first.

For your particular problem it seems it would be solved by continue 
running the other tests in the same module when a test has failed. I 
think "silly" [2] looks really interesting. I haven't had the time yet 
to try it out so I don't know if it will continue after a failed test.

[1] https://dlang.org/spec/traits.html#getUnitTests
[2] https://code.dlang.org/packages/silly

-- 
/Jacob Carlborg