D-thrift package detects regressions since 2.061, where is the regression suite located?

[H. S. Teoh]

On Fri, Aug 23, 2013 at 11:07:35AM -0700, Walter Bright wrote:
> On 8/23/2013 10:34 AM, Denis Shelomovskij wrote:
> >By the way, the ability to add costume projects to D autotester is
> >already proposed without any response:
> >http://forum.dlang.org/thread/kqm4ta$m7f$1@digitalmars.com
> >
> 
> The question comes up repeatedly, and I've answered it repeatedly,
> the latest on 8/20 in the thread "std.serialization: pre-voting
> review / discussion". Here's the message:
> ---------------------------------
> On 8/18/2013 9:33 AM, David Nadlinger wrote:
> > Having a system that regularly, automatically runs the test suites
> > of several larger, well-known D projects with the results being
> > readily available to the DMD/druntime/Phobos teams would certainly
> > help. But it's also not ideal, since if a project starts to fail,
> > the exact nature of the issue (regression in DMD or bug in the
> > project, and if the former, a minimal test case) can often be hard
> > to track down for somebody not already familiar with the code base.
> 
> That's exactly the problem. If these large projects are incorporated
> into the autotester, who is going to isolate/fix problems arising
> with them?
> 
> The test suite is designed to be a collection of already-isolated
> issues, so understanding what went wrong shouldn't be too difficult.
> Note that already it is noticeably much harder to debug a phobos
> unit test gone awry than the other tests. A full blown project that
> nobody understands would fare far worse.
> 
> (And the other problem, of course, is the test suite is designed to
> be runnable fairly quickly. Compiling some other large project and
> running its test suite can make the autotester much less useful when
> the turnaround time increases.)
> 
> Putting large projects into the autotester has the implication that
> development and support of those projects has been ceded to the core
> dev team, i.e. who is responsible for it has been badly blurred.

One idea that occurred to me is to put large external projects under a
separate tester, not bound to the core dmd/druntime/phobos autotesting,
but an independent tester that regularly checks out git HEAD and
compiles & tests said large projects. The devs can then monitor the
status of these tests independently, and when something goes wrong, they
can ping whoever is responsible for that project to investigate what
might be the cause. If it's caused by the latest git commit(s), they can
file regression bugs on the bugtracker, otherwise, they update their
code to work with the new compiler. If the responsible person doesn't
respond, or there is no contact person, we could use this forum as a
kind of community notice that something needs to be fixed somewhere.  If
nobody responds, the project is likely not worth keeping up with.

This way we don't slow down development / autotesting unnecessarily, and
still let the community know when there might be potential problems with
existing code. (It probably also helps for the core devs to be aware of
potential regressions without being held back from code changes.) If
it's important, *somebody* will step up and file bugs and/or fix the
issue. If nobody cares, then probably it's not worth fretting over.


T

-- 
Do not reason with the unreasonable; you lose by definition.