CURL Wrapper: Congratulations Next up: std.serialize

[Jonathan M Davis]

On Saturday, December 31, 2011 18:57:14 Andrew Wiley wrote:
> On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis <jmdavisProg at gmx.com> 
wrote:
> > On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
> >> On 2011-12-31 11:37, Jonathan M Davis wrote:
> >> > On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
> >> >>> I think that the AssertError's message (which includes the
> >> >>> file and
> >> >>> line number of the failure) and its stack trace are plenty.
> >> >>> It's
> >> >>> exactly what you need and nothing else.
> >> >>> 
> >> >>> - Jonathan M Davis
> >> >> 
> >> >> I want to have such a summary.
> >> > 
> >> > I don't see any reason to put that in the standard library.
> >> > There's
> >> > nothing wrong with 3rd party solutions which give additional
> >> > functionality, but D's unit test framework is designed to be
> >> > minimilistic, and I don't think that adding anything beyond what
> >> > it
> >> > does now in terms of summary makes any sense. The only major issue
> >> > in
> >> > that regard IMHO is the fact that no further unittest blocks
> >> > within a
> >> > module are run after one fails. Even if it did, I still don't
> >> > think
> >> > that a fancier summary would be worth having - especially in the
> >> > standard library.
> >> 
> >> BTW, what would be so wrong if the unit tests for the standard library
> >> displayed a nice report when finished?
> > 
> > My primary issue here is that I don't think that we should be adding
> > stuff to Phobos which is essentially a new unit test framework on top
> > of the built in one. If 3rd party stuff wants to do that. Fine. But the
> > standard library should use the standard facilities. If the standard
> > facilities aren't sufficient, then they should be improved.
> 
> The counterargument is that the language doesn't really provide a
> framework - it actually provides anonymous parameterless global
> functions that will be run before main is invoked if code is compiled
> with -unittest. That isn't considered a framework in any language I've
> ever used, but it adds just enough functionality to allow a
> well-integrated fully-featured library solution. Would making such a
> library solution part of the standard library really be a problem?
> I'm mostly ambivalent on this issue because I haven't had time to look
> closely at the proposed framework, but your argument seems to be that
> all unittesting functionality needs to be built into the language. I
> don't think that should be necessary or required.

For the most part, I don't think that we need anything more. It should be fixed 
so that all unittest blocks within a module are run even if one fails. This 
should be fixed in the compiler and runtime rather than having to hack it 
together with a library solution, since it's supposed to work that way in the 
first place but doesn't because of some changes that need to be made to the 
compiler (some hooks of some kind are needed IIRC). Named unit tests would 
also be nice, but it would be _far_ cleaner to add those to the language than 
try and add them with a library solution (e.g. unittest(test_name){}). It 
would also be nice if it were possible to hook in something that would run 
specific unit tests, but again, such hooks would then need to be provided by 
the compiler and/or runtime - unless you want to simply create a function for 
each unit test and have each unittest block call a specific function? That's an 
ugly solution IMHO.

All of those changes would be fairly minor in terms of how they affect how unit 
tests function and would be far cleaner IMHO to have in the language. I think 
that on some level, the unit test framework in the language has failed if we 
have to add library solutions on top of it to get such basic functionality.

> > As for a "nice report," I don't see anything wrong with just using the
> > stack traces (which include the file, line number, and error message of
> > the assertion failure). That's all the information that's needed.
> > Anything else is superfluous IMHO. Now, if there were something nicer
> > that could be generally agreed upon and added to druntime such that the
> > standard unit test facilities used it, then fine. I don't see any point
> > to it, but at least in that case, the standard library is still using
> > the standard unit test framework. What I really don't want to see is
> > Phobos essentially building a new unit test framework on top of the
> > existing one. Any issues that need to be addressed with the unit test
> > framework for the standard library should be addressed in the standard
> > framework. Any additional framework stuff should be left to 3rd
> > parties.
> 
> As I said above, I wouldn't consider what we have to be a framework,
> but it's definitely enough to build an excellent library solution on
> top of.
> As for a report, the problem is that an assertion error isn't what you
> want when you're running, say, a continuous integration server (or,
> say, a pull request tester). What you really want is a detailed
> explanation of what unittests broke, what the tests were testing, and
> how the result differed from what was expected. You want to be able to
> have a reasonable idea of what went wrong *without* having to look at
> someone else's code and figure out exactly what they're testing every
> time.

I don't really get what you'd need beyond knowing which test failed. The 
AssertErrors tell you that. It would definitely be nicer if the unittest blocks 
had proper names to go with them, but they're not necessary for knowing which 
test failed. The file and line number give you that. I don't see how you're 
going to know what actually failed unless you actually look at the test. If 
the tests have names and are well named, then that gives you a better idea of 
what isn't working before you look at the code, but ultimately, you have to 
look at the code to know what went wrong. And if you can name unittest blocks, 
then that problem is solved anyway, since then you'd have a name for the unit 
test failure to give in any report that you wanted to create.

- Jonathan M Davis