Thoughts about unittest run order

[H. S. Teoh]

In theory, the order in which unittests are run ought to be irrelevant.
In practice, however, the order can either make debugging code changes
quite easy, or very frustrating.

I came from a C/C++ background, and so out of pure habit write things
"backwards", i.e., main() is at the bottom of the file and the stuff
that main() calls come just before it, and the stuff *they* call come
before them, etc., and at the top are type declarations and low-level
functions that later stuff in the module depend on.  After reading one
of Walter's articles recently about improving the way you write code, I
decided on a whim to write a helper utility in one of my projects "right
side up", since D doesn't actually require declarations before usage
like C/C++ do.  That is, main() goes at the very top, then the stuff
that main() calls, and so on, with the low-level stuff all the way at
the bottom of the file.

It was all going well, until I began to rewrite some of the low-level
code in the process of adding new features. D's unittests have been
immensely helpful when I refactor code, since they catch any obvious
bugs and regressions early on so I don't have to worry too much about
making large changes.  So I set about rewriting some low-level stuff
that required extensive changes, relying on the unittests to catch
mistakes.

But then I ran into a problem: because D's unittests are currently
defined to run in lexical order, that means the unittests for
higher-level functions will run first, followed by the lower-level
unittests, because of the order I put the code in.  So when I
accidentally introduced a bug in lower-level code, it was a high-level
unittest that failed first -- which is too high up to figure out where
exactly the real bug was. I had to gradually narrow it down from the
high-level call through the middle-level calls and work my way to the
low-level function where the bug was introduced.

This is quite the opposite from my usual experience with "upside-down
order" code: since the low-level code and unittests would appear first
in the module, any bugs in the low-level code would trigger failure in
the low-level unittests first, right where the problem was. Once I fix
the code to pass those tests, then the higher-level unittests would run
to ensure the low-level changes didn't break any behaviour the
higher-level functions were depending on.  This made development faster
as less time was spent narrowing down why a high-level unittest was
failing.

So now I'm tempted to switch back to "upside-down" coding order.

What do you guys think about this?


T

-- 
You have to expect the unexpected. -- RL