workflows/eval: Add the eval summary as a comment

[azuwis]

See it in action azuwis#9

Make #360339 more discoverable.

Will post a comment like:

---

Eval summary

• Added packages: 0
• Removed packages: 0
• Changed packages: 1
• Rebuild Linux: 1
• Rebuild Darwin: 0

---

Compared to #360962:

1. Only post single comment, and update it on subsequent push, reduce noise for long-running pull requests
   See workflows/eval: Add the step summary as a comment #360962 (comment)
2. Only summary and link, no details, will not affect GH search results
   See workflows/eval: Add the step summary as a comment #360962 (comment)

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[azuwis]

@Mic92 @FliegendeWurst What do you think?

[FliegendeWurst]

I think the main benefit here is the link to the evaluation results. Numbers are already provided by the labels

[Mic92]

works: #361344 (comment)

[wolfgangwalther]

Is this going to be posted on every force push?

If yes: strong 👎

Edit: I see --edit-last is used. That should work.

[infinisil]

Hmm not sure if I like having such a comment on every single PR

[azuwis]

How about only post the comment, if more than 10 packages are added/removed/changed? Like this #361365 (comment)

[wolfgangwalther]

It would be great if this could somehow be integrated with the job requesting reviews from OWNERS. What I mean is that it would be great if the bot could post a comment before the reviews are requested and then would update it later on... that could reduce the number of notifications a bit? You'd only receive a notification for review once and not a second one for the comment a bit later.

[Mic92]

Does requesting reviews actually creates a notification? I don't think it did this for me in the past. I agree with @infinisil I noticed that I found myself having to open updated pull requests more often just to see it was only yet another bot comment: #361388

[wolfgangwalther]

Does requesting reviews actually creates a notification?

If that was not the case, then those review requests would be useless. I get emails like this:

@nix-owners[bot] requested your review on ...

And then a few minutes later another comment about the eval result - which I really don't care about at this stage. I might care about it for my own PRs, but also... not really. I would really like to see the evaluation results on a quick glance once I look at the PR anyway, but I don't want to be notified about them.