ZERO Hydra Failures 21.05

[jonringer]

Update

I have updated the steps to reflect the new backport workflow. Please review, if not familiar with the process.

Mission

Every time we branch off a release we stabilize the release branch.
Our goal here is to get as little as possible jobs failing on the trunk/master jobsets.
I'd like to heighten, while it's great to focus on zero as our goal, it's essentially to
have all deliverables that worked in the previous release work here also.

NOTE: ZHF for this release will be significantly different from past ZHFs, with the
changes included in RFC 85.
Most significantly, branch off will occur on May 21st; prior to that date, ZHF will be conducted
on master; after that date, ZHF will be conducted on the release channel using a backport
workflow similar to previous ZHFs.

Jobsets

trunk Jobset (includes linux, darwin, and aarch64-linux builds)
nixos/combined Jobset (includes many nixos tests)

How many failing jobs are there?

At the opening of this issue we have the main x86_64-linux jobset at 643 failing jobs, x86_64-darwin at 1251, and aarch64-linux at 1067.

Thanks to nix-review-tools we know which dependencies are causing the most jobs to fail in these individual jobsets:

• nixos/release-21.05
• nixos/release-21.05-aarch64
• nixpkgs:nixpkgs-21.05-darwin
• nixpkgs/staging-next-21.05

Previous releases first evals

19.09 had 1654 failing jobs
20.03 had 1204 failing jobs
20.09 had 1153 failing jobs
21.05 had 789 failing jobs
So we're actually getting better at maintaining a more stable "unstable" channel.

How to help (textual)

1. Select an evaluation of the trunk jobset
   

2. Find a failed job ❌️ , you can use the filter field to scope packages to your platform, or search for packages that are relevant to you.
   

3. Search to see if a PR is not already open for the package. It there is one, please help review it.

4. If there is no open PR, troubleshoot why it's failing and fix it.

5. Create a Pull Request with the fix targeting master, wait for it to be merged.
   If your PR causes around 500+ rebuilds, it's preferred to target staging to avoid compute and storage churn.

6. After your PR to master or staging has occured. Please follow backporting steps and target the release-21.05 branch if the original PR landed in master or staging-21.05 if the PR landed in staging. Be sure to do git cherry-pick -x <rev> on the commits that landed in unstable. I created a video covering the backport process.

Always reference this issue in the body of your PR:

ZHF: #122042

Please ping @NixOS/nixos-release-managers on the PR.
If you're unable to because you're not a member of the NixOS org please ping @jonringer

How can I easily check packages that I maintain?

You're able to check failing packages that you maintain by running:

# from root of nixpkgs
nix-build maintainers/scripts/build.nix --argstr maintainer <name>

New to nixpkgs?

I created some videos to help get people started with nixpkgs:

• Packaging a basic C application
• Python nix packaging
• Adding a package to nixpkgs

Also be sure to check out other resources at: https://github.com/nix-community/awesome-nix

Packages that don't get fixed

The remaining packages will be marked as broken before the release (on the failing platforms).
You can do this like:

meta = {
  # ref to issue/explanation
  # `true` is for everything
  broken = stdenv.isDarwin; 
};

Closing

This is a great way to help NixOS, and it is a great time for new contributors to start their nixpkgs adventure. 🥳

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers

Related Issues

• Timeline: Discourse: 21.05 Release Timeline
• Feature Freeze Items: #21.05 Feature Freeze #121972

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-21-05-zero-hydra-failures/12954/1

[Ma27]

just figured that it might make sense to pin this for easier discoverability :)

[jonringer]

@NixOS/release-engineers would you all be okay with me changing the wording to have you all pinged for each ZHF PR. I don't want to overload individuals

[jonringer]

Fixes can always be backported. Going to close this for now, as the release has already happened.

Thank you everyone for your hard work! :)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/21-05-retrospective/13426/5