unittests are really part of the build, not a special run

[Idan Arye via Digitalmars-d]

On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu 
wrote:
> This is a tooling issue.
>
> ...
>
> In brief: I'd like to transition to a model in which 
> unittesting is organically part of the build. After all, you 
> wouldn't want to deploy an application that's failing its 
> unittests.
>
> Detail: Consider running a build vs. a unittest in one of the 
> supported editors/IDEs (emacs, vim, Code:Blocks, Visual D, 
> Xamarin...). During a build, an error will come in a standard 
> format, e.g.
>
> std/array.d(39): Error: undefined identifier xyz
>
> This format is recognized by the editor and allows the user to 
> click on it and go straight to the offending line etc.
>
> In contrast, a failing unittest has a completely different 
> format. In fact, it's a format that's worse than useless 
> because it confuses the editor:
>
> core.exception.AssertError at std/array.d(39): xyz
>
> ...
>
> Where I want us to be is a place where unittests are considered 
> a part of the build; it should be trivial to set things up such 
> that unittesting is virtually indistinguishable from 
> compilation and linking.
>
> ...
>
> Andrei

There is no point in running unittests before `main` in the same 
executable, but that doesn't mean the build is the right place 
for running unittests. The IDE/build-system should be the one 
that handles running the tests(both unit and integration). The 
compiler should give the IDE/build-system enough tools to do it 
properly - not to do it for them. Running UTs as part of the 
build blocks some options, like building a UT executable and 
running it in another environment where building is not possible.

Ideally, I would like to see the compiler, when ordered to do a 
unittest build, create an executable that only runs unittests and 
ignores `main`. In matter of fact, what it does is run a special 
"ut-main" entry point function declared in Phobos(or in the 
runtime - whichever makes more sense) that runs all the 
unittests, and can possibly catch `AssertError`s and display them 
in proper format.

Since we no longer run `main`, the command line arguments can go 
to the ut-main function. This doesn't change anything with the 
built-in ut-main since it just ignores them, but if that special 
entry point can be overrideable via a compiler flag, and 
IDE/build-system can supply it's own, more complex ut-main that 
can make use of the command line arguments. That special ut-main 
can communicate with the IDE/build-system to provide a better UX 
for unittest running - for example an IDE could do a graphical 
display of the unittests run, when it's custom ut-main is 
responsible for providing it info about the ut-run progress.