This is why I don't use D.

[H. S. Teoh]

On Fri, Sep 07, 2018 at 09:24:13AM -0600, Jonathan M Davis via Digitalmars-d wrote:
[...]
> I don't see how such a system could work very well if it's not
> automated.  Simply doing it with every release of dmd means that newer
> packages won't be tested, and it would force someone to take the time
> to run stuff every dmd release, which they may or may not have time to
> do. IMHO, it really only makes sense to set up something automated.

Yes, it needs to be automated and run in the background by a machine.
Humans are just too unreliable for this.


> I would expect that any time code.dlang.org detected that a package
> had a new version (or a package was added), the new versions would be
> tested.

That would make the most sense.


> What's somewhat more of an open question is how new compiler releases
> should be handled. Aside from the issue that _every_ package then
> potentially needs to be tested (which could take a while), there's the
> issue of which versions to test. Testing every release of a package
> would be overkill, but simply testing the latest isn't necessarily
> enough.

The initial setup will take a long time because we have so many packages
that have never been tested in this way before.  But once the database
is up-to-date, I expect that it will take much less work to keep things
up-to-date.

For the initial setup, I don't think it's necessary to test *every*
historial release of the compiler -- the most important thing is the
current official releases of all compilers (gdc, ldc, dmd).[*]  If a
package doesn't compile with the latest release, then optionally do a
unidirectional search backwards to find the last compiler version that
compiles it successfully, and log that.

([*] I was going to suggest including dmd-nightly as well, but that
poses the problem of load: running it every night will cause a lot of CI
churn, which also generates a lot of mostly-useless information -- no
one will care about which of 50 dmd git revisions failed / succeeded to
compile a package, just whether the *latest* dmd-nightly works.  So
perhaps an update once a week or once a month in between releases will
be enough.  But dmd-nightly is an optional extra that can be skipped for
now.  The important baseline is the current official releases of gdc /
ldc / dmd.)


> Also, there's the question of what it even means to test each package
> to verify that it's working. Does that mean running the unit tests?
> Not all projects run them the same way, and it could be pretty
> expensive to run the tests for all packages on code.dlang.org. And of
> course, that assumes that the package even has any unit tests to begin
> with. So, should testing the package just mean that dub build works?
> That will catch the really basic problems, but it won't usually catch
> problems in templates and could easily miss other types of problems -
> though maybe it's enough. Regardless, it's going to need to be made
> clear what it means that code.dlang.org claims that a package works
> with a particular version of the compiler.
[...]

Let's not overcomplicate things and doom the flight before it takes off.
Let's start with the absolute minimum baseline, and then additional
perks can be added on top of that once the baseline is working.

I say the baseline is that dub build works.

If we feel like going an extra mile, dub test.

If we feel like doing even more, then add other stuff like compiling
provided example programs, testing template instantiations and what-not.
(Though one would expect that dub test ought to do that already,
otherwise I question the quality of the code / unittests.)

As long as we publish exactly what is being run to verify whether the
package "works", I think that should be good enough for starters.  Even
if dub build, say, doesn't catch all problems, it's still better than
the status quo of no information at all.  Let's not let the perfect
become the enemy of the good, as is the common malady around here.


T

-- 
The problem with the world is that everybody else is stupid.