18.09 Zero Hydra Failures

[samueldr]

Let's make Jellyfish the best release so far!

We have: the main jobset starting at 425 failures, x86_64-darwin at 829, and aarch64-linux at ~1120. The numbers may seem large, but one weird trick appropriate fix may fix many at once.

How can I help???

• Choose some package from those that fail on Hydra. Hurry before the good ones have been taken!
• Find a fix. That may mean simply restricting meta.platforms, in case the package inherently doesn't support what it has in there ATM.
• Typically the package was broken on master already. You can verify that on Hydra - example URL: https://hydra.nixos.org/job/nixpkgs/trunk/bash.x86_64-linux In that case base the fix on master and request backporting in the description of the pull request.
• Ping this issue from the PR, e.g. /cc ZHF 18.09 Zero Hydra Failures #45960. Do this also if you have some WIP. Alternatively you may just post a note in this issue. If the breakage is specific to darwin 18.09 Zero Hydra Failures (Darwin) #45961 or aarch64 18.09 Zero Hydra Failures (aarch64) #45962 mention the respective issue instead.

The remaining packages will be marked as broken before the release (on the failing platforms), i.e. at the end of September. /cc @NixOS/nixpkgs-committers, but everyone can help out!

[xeji]

Reverting ad47c38 will fix nixpkgs.perl*Packages.MouseXGetOpt

reverted in 9889c0f and 4c00a04

[samueldr]

Failures report as of right now.

• https://gist.github.com/72d1f646f89b61520168b099c7dcc00b

Let's see how useful this is as a format. This was queried from the last finished eval, there were evals running while this was made.

[xeji]

This was queried from the last finished eval, there were evals running while this was made.

@volth the table shows an earlier eval where UNIVERSALref wasn't marked as broken yet, see the build logs, it is fine in the latest eval: https://hydra.nixos.org/eval/1477017#tabs-removed

Or should each dependent have its own meta.broken = versionAtLeast perl.version "5.26" ?

No, only the package that is broken itself.

[vcunat]

I'd add that broken and other checks are transitive during evaluation (implemented as exceptions).

[timokau]

The sage failures are due to a mistake I made when adding pkg-config aliases to openblas and the recent numpy update. I fixed openblas in #46016 in staging. I don't know if that means it will also be merged into 18.09. I haven't gotten to backporting the numpy upgrade from sage upstream yet.

I really think it is a shame that hydra doesn't ping maintainers on failures anymore. Seems like an essential feature to miss.

[xeji]

@vcunat @samueldr there are a number of changes currently in staging/staging-next that should go to 18.09 once they reach master - openblas, texlive 2018, a systemd bugfix, etc.
What's the workflow for these? Guess we'll need a staging-18.09 branch + Hydra job.

[vcunat]

Since the fork point the staging branch won't get to 18.09 anymore. Cherry-picking should be done if desired. For this one I did it in 6f8e07a.

[vcunat]

@xeji: we have staging-18.09 already, I think we'll add a jobset soon as well to make ZHF easier. (People don't want to rebuild everything when fixing other packages.) This openblas commit wasn't a mass rebuild according to my measurements, so I pushed it directly.

Reminder: we should choose the changes a bit more carefully than for usual staging, so that real stabilization happens :-)

[xeji]

8fb90de (in staging-18.09) fixes a regression introduced with systemd 239, and makes the systemd test pass.
Update: Fix is in release-18.09 now, done.

[domenkozar]

I've released stack2nix 0.2.1 to address version bump, once haskell packages are synced it should compile again.

[xeji]

boost162 is fixed by c70ff28 and 11c2595 (in staging-18.09, mass rebuild)
Update: fixes are in release-18.09 now, done.

[NeQuissimus]

I removed the Copperhead kernel yesterday, so those kernel modules should no longer fail :) The whole kernel is unsupported and unmaintained.

[xeji]

sage is still broken on 18.09 (but not on master). ping @timokau

[timokau]

@xeji thanks for the heads up. I cannot reproduce a failure with the most recent release-18.09, so that failure is probably outdated.

[vcunat]

@timokau: the newest evaluation failed three times (on different Hydra machines): https://hydra.nixos.org/build/81780926 There's always some "N doctests failed" at the end.

[qknight]

you lack a implementation (library, egg) where fib is defined, see the error

NameError: name 'fib' is not defined

**********************************************************************
File "/nix/store/9nix1fnsj56iqqcjavkar8gbjnic8djm-sage-src-8.3/src/sage/repl/ipython_extension.py", line 403, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    fib
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_extension.SageMagics.fortran[3]>", line 1, in <module>
        fib
    NameError: name 'fib' is not defined
**********************************************************************
File "/nix/store/9nix1fnsj56iqqcjavkar8gbjnic8djm-sage-src-8.3/src/sage/repl/ipython_extension.py", line 407, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    fib(a, 10)
Exception raised:
    Traceback (most recent call last):
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_extension.SageMagics.fortran[6]>", line 1, in <module>
        fib(a, Integer(10))
    NameError: name 'fib' is not defined
**********************************************************************
File "/nix/store/9nix1fnsj56iqqcjavkar8gbjnic8djm-sage-src-8.3/src/sage/repl/ipython_extension.py", line 408, in sage.repl.ipython_extension.SageMagics.fortran
Failed example:
    a
Expected:
    array([  0.,   1.,   1.,   2.,   3.,   5.,   8.,  13.,  21.,  34.])
Got:
    array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
**********************************************************************

[timokau]

@vcunat I still cannot reproduce that locally. The main error seems to be

File "/nix/store/9nix1fnsj56iqqcjavkar8gbjnic8djm-sage-src-8.3/src/sage/misc/inline_fortran.py", line 65, in sage.misc.inline_fortran.InlineFortran.eval
Failed example:
    fortran(code, globals())
Exception raised:
    Traceback (most recent call last):
      ...
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/site-packages/sage/misc/inline_fortran.py", line 125, in eval
        raise RuntimeError("failed to compile Fortran code:\n" + log_string)
    RuntimeError: failed to compile Fortran code:
      ...
      File "/nix/store/0bg15dw9whi80i71ldkxkyh1v5ck74lv-python-2.7.15-env/lib/python2.7/threading.py", line 736, in start
        _start_new_thread(self.__bootstrap, ())
    thread.error: can't start new thread

With the rest being avalanche errors. Could that be some sort of hydra threading issue? Can anybody reproduce the error?

[samueldr]

Hmm, last times I wanted to generate a report there weren't good evals to check. Now there is:

• https://gist.github.com/d5d5b4915c4dc9cf126323188f3d48ad

It looks like a chunk of the new failures on aarch64 would be fixed with #47564.

@vcunat any blockers for a release you think? I don't have anything in mind, since staging was merged and the last things I knew of were all on staging.

While not strictly ZHF related, the blockers don't seem to be severe enough to warrant blocking the release? Most are either older than this release cycle, or things that don't really block, but still would be great to finalize.

[matthewbauer]

It sounds alright to me. Someone had mentioned merging #42846 before the release but since it's a nonessential module I would consider it nonblocking (although might be worthwhile backporting).

Make sure you look at the 18.09 milestone PRs though too: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+is%3Apr+milestone%3A18.09

[peterhoeg]

I consider #47577 a blocker - apologies for not raising it earlier.

[timokau]

It looks like the sage build succeeded, curious.

As for the release I suggest removing all the blocker labels and bumping the milestones, then give the people involved at least one day to complain before going ahead with 18.09.

[samueldr]

Hey, what do you know: I found a blocker, and I should have known better and known it is one beforehand:

See #47602, while this doesn't block on a technical side, this is a definite blocker on the human side, with a side dish of bad user experience as their first bite into NixOS.

[timokau]

In case anybody was on the edge of their seat because of the transient sage failure: Turns out it was caused by some numpy issue with high cpu count. Since I'm still seeing the issue in master, its probably still there in release-18.09 too. I've opened a PR (#49888) to cherry-pick the fix to numpy in release-18.09. For master it should be good enough to wait for the next numpy upgrade.

[cyounkins]

Looks like we had a huge bump at https://hydra.nixos.org/eval/1495549

Many are propagated from https://hydra.nixos.org/build/85961701 which failed with "Log limit exceeded" and "building of '/nix/store/5awxqywjwjldazlzls4jslgm1l828hb3-nbd-3.18' killed after writing more than 67108864 bytes of log output" despite not writing 6MB to the log in the web UI. Possible hydra issue? @grahamc

[vcunat]

Apparently someone has restarted these builds and they succeeded. (Yes, I'm a bit late.)