How can one reliably run unittests

[Steven Schveighoffer]

On 8/27/21 7:19 AM, Johan wrote:
> On Friday, 27 August 2021 at 10:59:54 UTC, Mathias LANG wrote:
>> On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
>>> On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
>>>>
>>>> But when I throw in the `-unittest` switch (or the `-cov` or 
>>>> `-profile` for that matter), I want all the bells and whistles. Give 
>>>> me colors, an ncurses interface, an HTTP server even! Give me the 
>>>> full battery pack. Don't expect everyone and their mother to 
>>>> implement their own testing framework, that's just bonkers.
>>>
>>> Why would you expect a _compiler_ to implement a testing framework?
>>
>> I don't. I expect the _runtime_ to do it. And to give me the tool to 
>> override it if I have needs that aren't covered by it. Currently it 
>> only does the later.
> 
> For this discussion I consider the compiler+runtime to be a single 
> entity, but OK: same question, why should the language runtime do it? I 
> find it a large stretch to include full unittesting framework inside the 
> scope of a language runtime's core functionality.
> What is so special about unittesting that requires the implementation to 
> be inside the runtime? The downsides are clear to me (and large), what 
> advantage is gained by having it in the language runtime?

separate compilation. The compiler doesn't know all the modules that 
have been built with unittests. Only the runtime does.

The compiler embeds unittests into the ModuleInfo, which means after the 
linking is done, then you can run unittests if there are any.

But the unittest "framework" right now is really simple (it's in fact 
[this 
function](https://github.com/dlang/druntime/blob/e6caaab9d359198b760c698dcb6d253afb3f81f6/src/core/runtime.d#L544-L655). 
I think it should stay that way, and if you want it more complex, use 
your own test system. I spent a good deal of time making it easier to 
install your own handler (in addition to NOT running main after 
unittests by default).

One thing that could be useful is a user-specified way to generate 
unittest metadata. Right now, you just get a single function pointer in 
ModuleInfo. You lose any UDAs or references to individual unittest 
functions. Allowing one to specify a metadata collector on unittests 
would be very useful in separating the unittest framework from the 
dependencies, and making it much easier to select a different one, 
regardless of your dependency preferences.

-Steve