unittests are really part of the build, not a special run

[Atila Neves via Digitalmars-d]

I actually thought about the whole "it should fail to build if 
any of the unit tests fail" idea 2 or 3 weeks ago, so this sounds 
good.

WRT to the error messages and their recognition by text editors, 
a _massive_ improvement would be compiler-assisted formatting of 
the assertion errors. This:

core.exception.AssertError at foo.d(2): Assertion failure

Is not useful when I wrote `assert(foo == 2)`. This, however, is:

tests.encode.testEncodeMoreThan8Bits:
     tests/encode.d:166 - Expected: [158, 234, 3]
     tests/encode.d:166 -      Got: [158, 234]


In Python, my favourite testing framework is py.test. It reflects 
on the test code itself and replaces `assert foo == 2` with its 
own code so that it looks like this in the output:

     def test_foo():
         foo = 5
>       assert foo == 2
E       assert 5 == 2

It also recognises things like `assert x in xs`, which is 
obviously handy. Since Walter has mentioned the "specialness" of 
assert before, maybe the compiler could recognise at least the 
most common kinds and format accordingly (assert ==, assert in, 
assert is null, assert !is null)?

The main reasons I wrote a unit testing library to begin with 
were:

1. Better error messages when tests fail
2. Named unit tests and running them by name
3. Running unit tests in multiple threads

I'm address 1. above, 2. has its own thread currently and AFAIK 
3. was only done my me in unit-threaded. There are other niceties 
that I probably won't give up but those were the big 3.

Atila

On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu 
wrote:
> This is a tooling issue.
>
> D builds and messaging are rigged to consider unittests as a 
> special build followed by a special run of an application.
>
> In brief: I'd like to transition to a model in which 
> unittesting is organically part of the build. After all, you 
> wouldn't want to deploy an application that's failing its 
> unittests.
>
> Detail: Consider running a build vs. a unittest in one of the 
> supported editors/IDEs (emacs, vim, Code:Blocks, Visual D, 
> Xamarin...). During a build, an error will come in a standard 
> format, e.g.
>
> std/array.d(39): Error: undefined identifier xyz
>
> This format is recognized by the editor and allows the user to 
> click on it and go straight to the offending line etc.
>
> In contrast, a failing unittest has a completely different 
> format. In fact, it's a format that's worse than useless 
> because it confuses the editor:
>
> core.exception.AssertError at std/array.d(39): xyz
>
> emacs will recognize the text as a filename and upon clicking 
> would ask the user to open the nonsense file 
> "core.exception.AssertError at std/array.d". This error line is 
> followed by a stacktrace, which is in no recognizable format. 
> It should be in the format of e.g. gcc remarks providing 
> additional information for an error, and again provide 
> file/line information so the user can click and see the call 
> stack in the source.
>
> Where I want us to be is a place where unittests are considered 
> a part of the build; it should be trivial to set things up such 
> that unittesting is virtually indistinguishable from 
> compilation and linking.
>
> This all is relatively easy to implement but might have a large 
> positive impact.
>
> Please chime in before I make this into an issue. Anyone would 
> like to take this?
>
>
> Andrei