[Issue 11783] Make std.datetime unittesting faster

[d-bugmail at puremagic.com]

https://d.puremagic.com/issues/show_bug.cgi?id=11783



--- Comment #12 from Andrei Alexandrescu <andrei at erdani.com> 2013-12-22 11:31:04 PST ---
(In reply to comment #9)
> I completely agree that 38 seconds is too long. That needs to be significantly
> reduced.

That's the spirit. Better yet, we should not have gotten to this point in the
first place. That's 1.5 orders of magnitude beyond the "holy frak" moment.

> However, what I don't want to do is reduce it by simply reducing the
> number of tests. Experience has shown me (and other developers who I've talked
> to agree with me - e.g. Steven Schveighoffer who did the Tango time stuff years
> ago) that date/time-related tests need to be thorough because of all of the
> corner cases, many of which are hard to know because they depend on quirks of
> the calculations being done.

Though the assertions are true in principle, I completely disagree to where
this has led std.datetime. This is not gene splitting or the Mars rover.
There's just too much brute force testing exercising the same code paths over
and over again, beyond all sense and reason. I think the "unittests are good"
argument has been essentially used as an excuse to pile everything and anything
on the account that more unittests lead to better unittesting.

> Maybe the number of tests will have to be reduced
> in order to get the time down, but if I can, I'd very much prefer to get the
> time down by optimizing the tests, which I believe can be done.

I think rethinking the approach to unittesting std.datetime from the ground up
would be much better for everyone.

> Maybe I'm too touchy on this subject and come across badly, for which I'm
> sorry.

I'm sure I'm the more guilty party. 

> But I believe very strongly that code in general (not just std.datetime)
> needs to have very thorough testing, and so I tend to object to attempts to
> reduce the number of tests of any code.

Again, this is a sensible statement but I think you have taken it to extremes
that are wholly inappropriate. This is the main point I'm trying to get across
with this bug report: just because X is good, one million X is not necessarily
one million times (or any) better. We are contemplating a net and major
negative gain here by overdoing what is fundamentally good.

(In reply to comment #10)
> FYI. I just tried commenting out all of the assertThrown unit tests in
> std.datetime, and the running time on my machine went from nearly 24 seconds to
> just under 4. So, while the other tests definitely need to be refactored and
> optimized (in which case, they'd take even less time), the ones that throw
> exceptions currently take up about 5/6th of the running time (and take up a far
> smaller portion than that of the total tests). So, optimizing those tests is
> definitely low hanging fruit.

Great, that's the spirit. I suggest you establish a budget - like 2 seconds -
and stick with it. That's some 10 times more that what would be most
appropriate, but still within reasonable bounds.

> If you'd like, since I do have a few days off for Christmas and can spend at
> least a little bit of time on D, I can spend some time over the next few days
> strategically disabling tests in order to improve how long std.datetime's tests
> take without losing a lot of tests, and then I can go back and refactor them
> properly in a month or two when I finally have time to focus on that. So, you'd
> get faster execution times but poorer testing in the short term, and then for
> the long term, I can sort out how to get good execution times without losing a
> lot of testing.

This is a volunteer project and it's not my place to tell anyone what to do and
when. Family, school, work come first. I'm glad we got to the point of
acknowledging the matter, and I want to make sure we don't repeat the mistake
and that we consider it worth fixing.

-- 
Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------