More CI woes

Sun Sep 20 06:35:22 UTC 2020

On Sunday, 20 September 2020 at 03:16:57 UTC, Andrei Alexandrescu 
wrote:
> Several CI pipelines are failing for 
> https://github.com/dlang/phobos/pull/7638, so I moseyed to what 
> I thought would be a simple one, the doc build:
>
> http://dtest.dlang.io/results/d42adbc33e821c4223291e64541d57ffbb5b3b0d/eb9e6443fd3231b8d086509d950283587ac6ee9e/build.log
>
> I spent some time trying to make heads and tails of it. A few 
> things that could be improved:
>
> * It's 2942 lines long. Most of these lines are just chaff, 
> e.g. do we really need to log every single cp operation, or 
> would a "Copied all files" line suffice?
>
> * The word '\Werror\W' appears in the log 25 times. Although I 
> assume most are to be ignored, some occur in quite dire 
> contexts, such as: "Error parsing type 'const(void function()* 
> function() pure nothrow @property @nogc)': Missing ')' for 
> 'const('". Is it possible to massage error messages known to be 
> benign?
>
> * Some of the command lines being run are printed like this:
>
> [2020-09-20 02:25:55.606583] dautotest: Running: 'timeout' 
> '1800' 'make' '-f' 'posix.mak' 
> 'DMD=/home/dtest/DAutoTest/work/build/bin/dmd' '-j' '8' 
> 'MODEL=64'
>
> Command lines are definitely something people would want to 
> copy and paste from the log. They should be printed like this:
>
> [2020-09-20 02:25:55.606583] dautotest: Running: 'timeout 1800 
> make -f posix.mak DMD=/home/dtest/DAutoTest/work/build/bin/dmd 
> -j 8 MODEL=64'
>
> The lines that carry actual information appear like this:
>
> [1msrc/dmd/frontend.d(142): [1;31mError: [mtemplate 
> [0;36m[m[1mstd[0;36m.[m[1malgorithm[0;36m.[m[1miteration[0;36m.[m[1meach[0;36m[m cannot deduce function from argument types [0;36m!([m[1maddGlobalIdent[0;36m)([m[1mconst[0;36m([m[1mstring[0;36m[]))[m, candidates are:
> [1m/dev/shm/dtest/work/repo/dmd/generated/linux/release/64/../../../../../phobos/std/algorithm/iteration.d(805): [1;31m       [m[0;36m[m[1meach[0;36m([m[1malias[0;36m [m[1mfun[0;36m = [m[1m"a"[0;36m, [m[1mRange[0;36m)([m[1mauto[0;36m [m[1mref[0;36m [m[1mRange[0;36m [m[1mrange[0;36m)[m
>   with [0;36m[m[1mfun[0;36m = [m[1maddGlobalIdent[0;36m,
>        [m[1mRange[0;36m = 
> [m[1mconst[0;36m([m[1mstring[0;36m[])[m
>   must satisfy one of the following constraints:
> [0;36m       [m[1misInputRange[0;36m![m[1mRange[0;36m
>        [m[1misStaticArray[0;36m![m[1mRange[0;36m
>        [m[1mhasMember[0;36m!([m[1mRange[0;36m, 
> [m[1m"opApply"[0;36m)[m
>
> They have escape sequences for pretty printing with colors. 
> Whatever system is producing that output should detect it's not 
> running in a console and omit the escape sequences.
>
> Thanks in advance!

Well, your PRs break the world and this time the CIs actually 
manage to catch it.
Have a look at the Buildkite output of your PR: 
https://buildkite.com/dlang/phobos/builds/3903 where it breaks 
the build of many real-world projects:

---
make[1]: Entering directory 
'/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize'
---> Running tsv-summarize unit tests
dmd 
-I/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/utils.d /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/numerics.d /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/fieldlist.d /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/getopt_inorder.d /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/unittest_utils.d /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/common/src/tsv_utils/common/tsvutils_version.d  -unittest -main -run /var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(815): Error: template `std.algorithm.iteration.each` cannot deduce function from argument types `!((x) => x.processHeaderLine(lineFields))(DList!(Operator))`, candidates are:
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/distribution/bin/../imports/std/algorithm/iteration.d(805):        `each(alias fun = "a", Range)(auto ref Range range)`
   with `fun = __lambda2,
        Range = DList!(Operator)`
   must satisfy one of the following constraints:
`       isInputRange!Range
        isStaticArray!Range
        hasMember!(Range, "opApply")`
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(842): Error: template instance `tsv_utils.tsv_summarize.SummarizerBase!(BufferedOutputRange!(File))` error instantiating
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(559):        instantiated from here: `NoKeySummarizer!(BufferedOutputRange!(File))`
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(815): Error: template `std.algorithm.iteration.each` cannot deduce function from argument types `!((x) => x.processHeaderLine(lineFields))(DList!(Operator))`, candidates are:
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/distribution/bin/../imports/std/algorithm/iteration.d(805):        `each(alias fun = "a", Range)(auto ref Range range)`
   with `fun = __lambda2,
        Range = DList!(Operator)`
   must satisfy one of the following constraints:
`       isInputRange!Range
        isStaticArray!Range
        hasMember!(Range, "opApply")`
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(842): Error: template instance `tsv_utils.tsv_summarize.SummarizerBase!(Appender!(char[]))` error instantiating
/var/lib/buildkite-agent/builds/ci-agent-f16edc43-2dc3-4961-b9ae-3fc8efb9b382-4/dlang/phobos/build/eBay-tsv-utils/tsv-summarize/src/tsv_utils/tsv-summarize.d(1091):        instantiated from here: `NoKeySummarizer!(Appender!(char[]))`
../makeapp.mk:48: recipe for target 'unittest' failed
---

> * It's 2942 lines long. Most of these lines are just chaff,

Well, extra verbosity is nice when things fail.
Why don't you scroll to the bottom?

> I spent some time trying to make heads and tails of it.

Huh, it yields a fairly nice nice error message: (it's from the 
link you posted)

---
The following operation failed:
Dependencies:
  -> dmd
  -> 
/dev/shm/dtest/work/repo/dmd/generated/linux/release/64/VERSION
  -> 
/dev/shm/dtest/work/repo/dmd/generated/linux/release/64/SYSCONFDIR.imp

Sources:
  -> /dev/shm/dtest/work/repo/dmd/src/dmd/frontend.d

Targets:
  -> 
/dev/shm/dtest/work/repo/dlang.org/web/phobos-prerelease/dmd_frontend.html

Command: 
/dev/shm/dtest/work/repo/dmd/generated/linux/release/64/dmd -o- 
-c -Dd/dev/shm/dtest/work/repo/dlang.org 
-J/dev/shm/dtest/work/repo/dmd/src/dmd/res 
-I/dev/shm/dtest/work/repo/dmd/src/dmd 
/dev/shm/dtest/work/repo/dmd/src/project.ddoc 
-Df/dev/shm/dtest/work/repo/dlang.org/web/phobos-prerelease/dmd_frontend.html src/dmd/frontend.d -version=MARS -w -de -fPIC -m64 -J/dev/shm/dtest/work/repo/dmd/generated/linux/release/64 -I/dev/shm/dtest/work/repo/dmd/src -dip25 -g -color=on
---

> Command lines are definitely something people would want to 
> copy and paste from the log. They should be printed like this:

I am sorry, but I don't see your point.
1) The command as run must be displayed (no modification)
2) They actually work as displayed. Try `'timeout' '2s' 'sleep' 
'5s'` in your shell.

> > They have escape sequences for pretty printing with colors.
> Whatever system is producing that output should detect it's not 
> running in a console and omit the escape sequences.

https://issues.dlang.org/show_bug.cgi?id=21266