Extending unittests [proposal] [Proof Of Concept]

[Jens Mueller]

Johannes Pfau wrote:
> Am Fri, 21 Sep 2012 16:37:37 +0200
> schrieb Jens Mueller <jens.k.mueller at gmx.de>:
> 
> > Jacob Carlborg wrote:
> > > On 2012-09-21 14:19, Jens Mueller wrote:
> > > 
> > > >Why do you need filename and line information of a unittest. If a
> > > >unittest fails you'll get the relevant information. Why do you
> 
> With the recent name mangling change it's possible to get the unittest
> line if a test fails, but only if you have working backtraces. That
> might not be true for other compilers / non x86 architectures.
> 
> To get the filename you have to demangle the unittest function name
> (IIRC core.demangle can't even demangle that name right now) and this
> only gives you the module name (which you could also get using
> moduleinfo though)

I'm saying I do not care which unittest succeeded. All I need that all
unittest I ran where successfully executed.

> It's also useful for disabled tests, so you can actually look them up.

That may be useful. So you say these tests where disabled instead of
just 2 tests where disabled.

> > > >want the information when a unittest succeeded? I only care about
> > > >failed unittests. A count of the number of executed unittests and
> > > >total number is enough, I think.
> 
> The posted example shows everything that can be done, even if it might
> not make sense. However printing successful tests also has a use case:
> 
> 1: It shows the progress of unit testing. (Not so important)
> 2: If code crashes and doesn't produce a backtrace, you still now which
> test crashed as the file name and line number are printed before
> running the test. (might sound unprobable. But try porting gdc to a new
> architecture. I don't want to waste time anymore commenting out
> unit tests to find the failing one in a file with dozens of tests and
> an ARM machine that takes ages to run the tests)

Why don't you get report when the program crashes?

> Another use case is printing all unittests in a library. Or a gui app
> displaying all unittests, allowing to only run single unittests, etc.

Listing on a unittest level and selecting may be useful.

> > > But others might care about other things. I doesn't hurt if the
> > > information is available. There might be use cases when one would
> > > want to display all tests regardless of if they failed or not.
> > 
> > If there are use cases I agree. I do not know one.
> > The question whether there are *tools* that report in case of success
> > is easier to verify. Do you know any tool that does reporting in case
> > success? I think gtest does not do it. I'm not sure about JUnit.
> 
> I don't know those tools, but I guess they have some sort of progress
> indicator?

They have them at test case level. I'm not sure whether there is a
strict relation between unittest and test case for D. The module level
may be enough.

> But I remember some .NET unit test GUIs that showed a green button for
> successful tests. But it's been years since I've done anything in .NET.
> 
> > But of course if a unittest has additional information and that is
> > already implemented or easy to implement fine with me. My point is
> > more that for the common cases you do not need this. Maybe in most.
> > Maybe in all.
> 
> You usually don't have to print sucessful tests (although sometimes I
> wasn't sure if tests actually run), but as you can't know at compile
> time which tests fail you either have this information for all tests or
> for none.

But you could just count the number and report it. If it says
"testing std.algorithm with 134 of 134 unittest"
you know all have been executed. What is true it won't tell you which
unittests were disabled. But that is easy to find out.

> The main reason _I_ want this is for gdc: We currently don't run the
> unit tests on gdc at all. I know they won't pass on ARM. But the unit
> tests error out on the first failing test. Often that error is a
> difficult to fix backend bug, and lots of simpler library bugs are
> hidden because the other tests aren't executed.

But this is a different problem. You want to keep executing on failure.
You don't need a unittest name for this. Maybe you say skipping a
failing unittest is better and disabling them in the source using
@disable is tedious.

> I'm actually kinda surprised the feedback on this is rather negative. I
> thought running unit tests individually and printing line/file/name was
> requested quite often?

Running unittests individually is very useful. But I'm not so sure about
the latter. I think driving the execution of how to execute the
unittests is important. Not so much reporting listing single unittests.
But I won't object when you add this feature if you believe it will be
used. Just saying I have less use for it. And if the change is simple
it should be unlikely to introduce any bugs.

Jens