Thoughts about unittest run order

[Jonathan M Davis]

On Monday, May 6, 2019 12:13:37 PM MDT H. S. Teoh via Digitalmars-d wrote:
> In theory, the order in which unittests are run ought to be irrelevant.
> In practice, however, the order can either make debugging code changes
> quite easy, or very frustrating.
>
> I came from a C/C++ background, and so out of pure habit write things
> "backwards", i.e., main() is at the bottom of the file and the stuff
> that main() calls come just before it, and the stuff *they* call come
> before them, etc., and at the top are type declarations and low-level
> functions that later stuff in the module depend on.  After reading one
> of Walter's articles recently about improving the way you write code, I
> decided on a whim to write a helper utility in one of my projects "right
> side up", since D doesn't actually require declarations before usage
> like C/C++ do.  That is, main() goes at the very top, then the stuff
> that main() calls, and so on, with the low-level stuff all the way at
> the bottom of the file.
>
> It was all going well, until I began to rewrite some of the low-level
> code in the process of adding new features. D's unittests have been
> immensely helpful when I refactor code, since they catch any obvious
> bugs and regressions early on so I don't have to worry too much about
> making large changes.  So I set about rewriting some low-level stuff
> that required extensive changes, relying on the unittests to catch
> mistakes.
>
> But then I ran into a problem: because D's unittests are currently
> defined to run in lexical order, that means the unittests for
> higher-level functions will run first, followed by the lower-level
> unittests, because of the order I put the code in.  So when I
> accidentally introduced a bug in lower-level code, it was a high-level
> unittest that failed first -- which is too high up to figure out where
> exactly the real bug was. I had to gradually narrow it down from the
> high-level call through the middle-level calls and work my way to the
> low-level function where the bug was introduced.
>
> This is quite the opposite from my usual experience with "upside-down
> order" code: since the low-level code and unittests would appear first
> in the module, any bugs in the low-level code would trigger failure in
> the low-level unittests first, right where the problem was. Once I fix
> the code to pass those tests, then the higher-level unittests would run
> to ensure the low-level changes didn't break any behaviour the
> higher-level functions were depending on.  This made development faster
> as less time was spent narrowing down why a high-level unittest was
> failing.
>
> So now I'm tempted to switch back to "upside-down" coding order.
>
> What do you guys think about this?

I've run into this a number of times. I generally put the public stuff at
the top of the file and the private stuff at the bottom. and as an extension
of that, I tend to end up the higher level functions higher up in the file,
and the lower level functions lower in the file, potentially triggering the
exact problem you're describing. My solution is generally to just start
commenting out a bunch of code when I start having test failures with the
higher level functions.

In general, I've found that these sort of problems don't come up much when
first writing code (rather, it's usually a problem when refactoring). So, I
don't usually think about this problem when writing code, and honestly, it
would feel really backwards to me to put the lower level and private stuff
at the top of the file rather than the bottom, so I just put up with the
problem when it comes up. Maybe if I were hitting it all the time, I'd
rethink how I organize code, but it hasn't been a big enough problem for me
to want to change my approach, and commenting out code generally works quite
well when it does come up. Worst case, I copy the code I'm working on into
another file until it starts working again.

A similar problem is when I work on something in Phobos that tons of stuff
depends on (e.g. std.utf), which can often result in either tests failing
elsewhere in Phobos or uild failures, and in such cases, I usually just copy
the code elsewhere and get it working before updating the real files.

- Jonathan M Davis