How can one reliably run unittests

[Atila Neves]

On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:
> D's unittest block is a great idea. However, I find that in 
> practice, I fail to make good use of them and have been 
> neglecting them. Sometime by using an external test suite 
> instead, sometime neglecting testing altogether. How come?
>
> Well, the main reason is that, as far as I can tell, it is near 
> damn impossible to run them at scale in any sensible way. Let 
> me explain.
>
> Ideally, one would want a module to be a unit. Declaration, 
> implementation, but also tests are part of the unit. During the 
> dev cycle, it is desired to be able to runt he test on modules 
> to verify that everything works as advertised. While doing so 
> is easy on a pet project,a s the projects grows, contains 
> several libraries and executable, doing this become very 
> challenging.
>
> It is not really possible to build a monster executable that 
> contains everything. First, this would kill build times, but, 
> with several main around it won't link. So let's split into 
> component.
>
> Some of these component are libraries, some are executable. 
> Adding a main for libraries is required, or it won't link, but 
> adding one to executable is going to cause a link error. This 
> in itself is a major main in the ass, because that means there 
> is no one consistent way to unittest a module without knowing 
> if that module has a main. In addition, everything needs to be 
> built twice now, which is really undesirable.
>
> Maybe one could rdmd each modules to runt he tests? That seems 
> like the best approach to me, considering one doesn't want to 
> actually produce an artifact for the tests, simply run them. 
> this also has the main/no main problem, but in addition, it 
> often fails with incomprehensible linker errors. For some 
> reason, rdmd is not able to include all dependencies 
> consistently, but, even better, it doesn't take the same 
> standard flags as other linkers do, so it is not possible to 
> reuse existing build infrastructure to feed all the correct 
> flags to rdmd.
>
> This may seems like it wouldn't be that big of a deal if you 
> manage all your flags by yourself, but very quickly turns into 
> a nightmare once you have to use 3rd party libraries that ship 
> with their own set of flags.
>
> At this point, I would just wish that one could rdmd --unittest 
> modulename.d and just pass it a couple of -I, -L and -l flags 
> and have all of it work. I have no idea how to achieve that.
>
> This kind of orthogonality is important as a project scale. One 
> cannot just doctor all the unitests call to all the modules. 
> You need to be able to tell your build system something akin to 
> "hey, every time you encounter a D module, please implicitly 
> add this unitest target. Thanks." This cannot be made to work 
> if running unit tests is done in a way that depends on the 
> content of the module.
>
> I don't think my case is isolated, every single substantial D 
> project I encountered has some kind of test suite or something 
> similar instead of relying on the unitests within modules. This 
> is frankly a failure, and it is 100% due to poor UX as the 
> language feature itself is great.
>
> If someone has some bandwidth, it is IMO a high leverage task 
> that solve real problem for real people and 90% of the work is 
> already there. The feature is there, the runtime support &al is 
> there. just provide a consistent UX so its use can be 
> integrated properly in larger systems.
>
> Thanks in advance.

Interesting.

My ideal scenario wouldn't involve linkers at all - I don't want 
to spend time waiting for my code to link, especially since, the 
way I write code means that unit tests never need optimising or 
native code generated. I'd go as far as to say that if one's unit 
tests *do* need any of that, they need to rethink their 
mini-integra... err, unit tests. But I digress.

The way I'd like it to work personally would be to CTFE unit 
tests in a module and have them run whenever I save the file. No 
linker, no main function, just that.

Back to the issue at hand. The problem with wanting to pass flags 
to the rdmd is that unless the project is trivial it'll have dub 
dependencies, so doing that manually is a no-no. This is the same 
thing that happens all the time in compile-as-you-type plugins, 
and why I maintain two separate emacs packages to make that done 
automatically (one for CMake/C++ and one for dub/D). Which means 
asking dub. Which means a build system.

Which means you just gave me an idea for an awesome reggae rule 
to generate a library with all of the dependencies of a module, a 
main function to run the tests, and then voilà. It'd have to do 
this for each module, and the first time said all-dependencies 
library is built it'd take time, but after that it'd work.

You'd still have to wait for the linker though, and if for some 
reason you're not using lld (or worse, you're on Windows), the 
wait will be looooong.