rustPlatform.buildRustPackage: support `finalAttrs` style #194475

amesgen · 2022-10-04T19:43:11Z

Description of changes

This PR explores supporting the new finalAttrs derivation style for buildRustPackage, similar to what is suggested in #119942 (comment). It is basically an extended version of #179392, and in particular also would close #107070.

A few notes:

This change should be entirely opt-in and cause no rebuilds.
This PR adds an ad-hoc way to prevent some attrs (called "intermediate args" here) to be passed to the underlying derivation call by specifying removeFromBuilderEnv. Another option would be to allow an arbitrary function (similar to apply in the module system), but that might be too powerful/confusing. Also see rustPlatform.buildRustPackage: support finalAttrs style #194475 (comment).
Rust packages can be incrementally converted and then enjoy the benefits outlined in stdenv.mkDerivation: overlay style overridable recursive attributes #119942

As an example, I converted difftastic to the new style, such that changing its version is now much DRYer:
```
pkgs.difftastic.overrideAttrs (oldAttrs: {
  version = "0.37.0";
  src = oldAttrs.src.overrideAttrs (_: {
    outputHash = "sha256-LwDoIvhZj/1fHg2eWgadwTcegeOKPpY8aCAugLfKtDE=";
  });
  cargoHash = "sha256-j7PVzGCButQhxVXVrtWhT6a6F1SLNe0jAy8oGqr9NvQ=";
  cargoLock = null;
  postPatch = "";
})   
```
Also note that passthru.tests.version then automatically points to the new package.
The "overlay" in buildRustPackage has a few curiosities, i.e. we have to be careful to avoid infinite recursion errors:
- It is not possible to use the { a ? "", ...}@finalAttrs syntax. Even {...}@finalAttrs causes infinite recursion.
- Asserts involving attributes from finalAttrs have to be "hidden" inside of values. Right now, I moved them to the respective attributes/variables; another option would be to put them into a dedicated asserts attribute, i.e. asserts = assert (all asserts here); "";.
Maybe there are better solutions for these problems around, but they don't seem too terrible.

Things done

farcaller · 2022-10-12T11:30:48Z

I came to this from #107070 and I must say that the fact that I can actually see and fix the cargoSha256 hash mismatches in overrides is a major UX improvement.

ghost

Hey, I'm totally in favor of this PR, but the way it's written makes it extremely difficult to review!

Could you please search-and-replace finalAttrs with args and previousAttrs with args? That alone will cut the size of the diff in half.

Then please add a let args = { src = args.src or null; ... } to rebind (shadow) args. That will cut the diff down to only a dozen lines changed or so -- way easier to review.

Thanks,

ghost · 2022-12-05T22:59:07Z

pkgs/stdenv/generic/make-derivation.nix

       "checkInputs" "installCheckInputs"
       "__darwinAllowLocalNetworking"
       "__impureHostDeps" "__propagatedImpureHostDeps"
-       "sandboxProfile" "propagatedSandboxProfile"])
+       "sandboxProfile" "propagatedSandboxProfile"
+       "intermediateArgs"] ++ (attrs.intermediateArgs or [])))


Could we pick a more descriptive name here? Once chosen, it's hard to change this.

The purpose of this is to indicate which attributes should not become part of the builder's environment. So we should pick a name that makes that purpose obvious to the reader without having to open make-derivation.nix in order to figure it out. Something like removeFromBuilderEnv or filterFromDerivation or notEnvVars or something else entirely.

Good point, will go with removeFromBuilderEnv for now 👍

ghost · 2022-12-05T23:00:08Z

pkgs/build-support/rust/build-rust-package/default.nix

-    -> !(args ? cargoSha256 && args.cargoSha256 != null) && !(args ? cargoHash && args.cargoHash != null)
-    -> throw "cargoSha256, cargoHash, cargoVendorDir, or cargoLock must be set";
-assert buildType == "release" || buildType == "debug";
+let rustOverlay =


Technically an overlay is a list of overrides. This is just an override.

According to the nixpkgs manual:

Overlays are Nix functions which accept two arguments, conventionally called self and super, and return a set of packages.

rustOverlay doesn't fulfill the "set of packages" criterion, so using sth more vague seems sensible, will change 👍

ghost · 2022-12-05T23:06:40Z

pkgs/build-support/rust/build-rust-package/default.nix

+      src = finalAttrs.src or "";
+      srcs = finalAttrs.srcs or null;
+      sourceRoot = finalAttrs.sourceRoot or null;
+      unpackPhase = finalAttrs.unpackPhase or null;
+      cargoUpdateHook = finalAttrs.cargoUpdateHook or "";
+      # Name for the vendored dependencies tarball


In the future, for big refactors like this, it would make them easier to review if you "rebuilt" the args attrset up at the top of the file (taking values from finalAttrs instead of args of course). Then you could make tiny easy-to-approve changes to lines like this, simply inserting (argsRebuilt) after inherit.

Yeah, you raise a good point here and in #194475 (review) above. I can rewrite it in the style you are suggesting here, as there are no existing review threads etc. that would be invalidated by this. WDYT?

Your call. The cleanup in your d16ba39 was enough that I was able to get through the diff without losing track of what was going on.

Ok, then I will leave it is as is for now (unless someone else objects) -- I ran into difficult-to-debug "infinite recursion" errors when trying to refactor this, which I like to avoid debugging unless necessary 😄

ghost

+2 We should merge this. I'd like to use it in #204694 and elsewhere.

Ping @roberth (who introduced mkDerivation (finalAttrs:) and @Ericson2314 (the build-support/rust guru) to see if they have any objections.

This produced no post-eval changes on either OfBorg or my system (which has a lot of overlays) for x86+arm+powerpc full workstation, mips router-only. So I feel confident it won't break things. If it does you can blame me.

nixos-discourse · 2022-12-27T13:20:07Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1607

winterqt · 2022-12-27T20:07:56Z

Poke @roberth ^^

SuperSandro2000 · 2022-12-27T20:40:51Z

pkgs/build-support/rust/build-rust-package/default.nix

+  buildAndTestSubdir = previousAttrs.buildAndTestSubdir or null;
+
+  cargoBuildType =
+    assert buildType == "release" || buildType == "debug";


Can we move the assserts after the inputs? thats a bit cleaner than inlining them.

Due to the evaluation model of finalAttrs, the assertions can not be kept in the place they were before. Citing from the PR description:

Asserts involving attributes from finalAttrs have to be "hidden" inside of values. Right now, I moved them to the respective attributes/variables; another option would be to put them into a dedicated asserts attribute, i.e. asserts = assert (all asserts here); "";.

pkgs/build-support/rust/build-rust-package/default.nix

roberth

Just small suggestions; already looks good.

roberth · 2022-12-28T11:39:42Z

pkgs/stdenv/generic/make-derivation.nix

       "checkInputs" "installCheckInputs"
       "__darwinAllowLocalNetworking"
       "__impureHostDeps" "__propagatedImpureHostDeps"
-       "sandboxProfile" "propagatedSandboxProfile"])
+       "sandboxProfile" "propagatedSandboxProfile"
+       "removeFromBuilderEnv"] ++ (attrs.removeFromBuilderEnv or [])))


pkgs/build-support/rust/build-rust-package/default.nix

nixos-discourse · 2023-01-14T17:52:46Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1704

jtojnar · 2023-04-12T07:06:53Z

Resolved the conflict in master...jtojnar:nixpkgs:buildRustPackage-finalAttrs

SuperSandro2000 · 2023-04-12T08:50:52Z

pkgs/tools/text/difftastic/default.nix

  pname = "difftastic";
  version = "0.46.0";

  src = fetchFromGitHub {
    owner = "wilfred";
-    repo = pname;
-    rev = version;
+    repo = finalAttrs.pname;


Suggested change

repo = finalAttrs.pname;

repo = "difftastic";

this is a bit problematic because if someone appends something to pname like -unwrapped than the download link silently breaks because of FODs.

It's worse: it wouldn't even break for you but for someone else, because - having the output - you don't need to re-run the FOD.

D.R.Y: do repeat yourself ;)

This is useful to remove intermediate/temporary arguments used in `finalAttrs`-style derivations.

Co-authored-by: Sandro <[email protected]> Co-authored-by: Robert Hensing <[email protected]>

amesgen · 2023-04-12T16:34:27Z

pkgs/build-support/rust/build-rust-package/default.nix

+      # TODO using previousAttrs here as we otherwise trigger rebuilds for all
+      # FOD fetcher users as finalAttrs.postUnpack is prefixed below.
+      postUnpack = previousAttrs.postUnpack or null;


Side effect of #221093. This is not ideal; I guess one solution would be to accept the rebuilds, but maybe there is a different approach here.

Attributes with null values are completely removed from the .drv, so this doesn't look like a mass rebuild to me.

Here is a concrete example:

With this PR in its current state (48e9de7) as well as before this PR (fe2ecaf):

$ nix-instantiate -A cargo /nix/store/1977acij72524vy0c8ij19ng7nzrf6yv-cargo-1.68.2.drv

After applying this diff

diff --git a/pkgs/build-support/rust/build-rust-package/default.nix b/pkgs/build-support/rust/build-rust-package/default.nix index f8e1eef2e93..6ca8a730d59 100644 --- a/pkgs/build-support/rust/build-rust-package/default.nix +++ b/pkgs/build-support/rust/build-rust-package/default.nix @@ -58,7 +58,7 @@ let unpackPhase = finalAttrs.unpackPhase or null; # TODO using previousAttrs here as we otherwise trigger rebuilds for all # FOD fetcher users as finalAttrs.postUnpack is prefixed below. - postUnpack = previousAttrs.postUnpack or null; + postUnpack = finalAttrs.postUnpack or null; cargoUpdateHook = finalAttrs.cargoUpdateHook or ""; # Name for the vendored dependencies tarball name = finalAttrs.cargoDepsName or finalAttrs.name or "${finalAttrs.pname}-${finalAttrs.version}";

we get

$ nix-instantiate -A cargo /nix/store/3msgx9jda9kc6hvfl7vgf9b8nsbm5g6y-cargo-1.68.2.drv

Diff:

$ nix-diff /nix/store/1977acij72524vy0c8ij19ng7nzrf6yv-cargo-1.68.2.drv /nix/store/3msgx9jda9kc6hvfl7vgf9b8nsbm5g6y-cargo-1.68.2.drv - /nix/store/1977acij72524vy0c8ij19ng7nzrf6yv-cargo-1.68.2.drv:{out} + /nix/store/3msgx9jda9kc6hvfl7vgf9b8nsbm5g6y-cargo-1.68.2.drv:{out} • The input derivation named `cargo` differs - /nix/store/qcsj2rh9k0dgig555kb12vx6zrzsw2nr-cargo.drv:{out} + /nix/store/fmvz39xbarfgssly8j6lzjp5dcxpgwxb-cargo.drv:{out} • The input derivation named `cargo-auditable-0.6.1` differs - /nix/store/7gkiqcx8qigih6pzga18ps6w5dpv3n33-cargo-auditable-0.6.1.drv:{out} + /nix/store/lnxzsh3ij88gp0vzj0628p1dn4gd2irj-cargo-auditable-0.6.1.drv:{out} • The input derivation named `cargo-auditable-0.6.1-vendor.tar.gz` differs - /nix/store/n67n6kbyd0ch4v88wwra9fy57p6mr263-cargo-auditable-0.6.1-vendor.tar.gz.drv:{out} + /nix/store/wkvcmy569xq0lgwbfvismc7f1lwb8qhb-cargo-auditable-0.6.1-vendor.tar.gz.drv:{out} • The environments do not match: + postUnpack=eval "$cargoDepsHook" export RUST_LOG= • Skipping environment comparison • Skipping environment comparison • Skipping environment comparison

Does that demonstrate the problem more clearly?

9999years · 2024-02-13T17:21:58Z

Is there anything keeping this from getting merged? It looks like the last review was almost a year ago. It's not clear which of the comments are blocking and which aren't. This functionality is widely requested and we should take it over the line, either by merging this PR or buildRustPackage: add overrideRust attribute #288430.
I'm not entirely sure of the semantics with this approach. I noticed that the examples use overrideAttrs — does that replace/remove the existing overrideAttrs functionality?

amesgen · 2024-02-13T17:43:52Z

Is there anything keeping this from getting merged? It looks like the last review was almost a year ago. It's not clear which of the comments are blocking and which aren't.

I think there are no fundamental blockers, but the code here is rather tricky as it is super easy to accidentally write sth that causes infinite recursions, see #194475 (comment). I hope to rebase it in the next days, but everyone is also welcome to take over this PR if they want to drive this.

I stopped regularly rebasing this PR as I wanted to wait for the pkgs-modules Nixpkgs Architectore working group, but I just saw that it was shut down for now.

This functionality is widely requested and we should take it over the line, either by merging this PR or #288430.

From my PoV, #288430 could be merged before this PR as it is useful as long as some Rust packages are not yet using the finalAttrs style.

I'm not entirely sure of the semantics with this approach. I noticed that the examples use overrideAttrs — does that replace/remove the existing overrideAttrs functionality?

It is the regular overrideAttrs, this PR just enables to use the advantages that the finalAttrs style (#119942) brings to overrideAttrs, also see https://nixos.org/manual/nixpkgs/unstable/#sec-pkg-overrideAttrs.

nikstur · 2024-05-14T07:55:14Z

Having this feature would really make my life much easier. It would make this override that I use in bombon to add a passthru SBOM to a Rust derivation much simpler: https://github.com/nikstur/bombon/blob/main/nix/passthru-vendored.nix#L9

Also it would make it much easier to add a passthru clippy or rustfmt derivation directly from the same file.

nixos-discourse · 2024-06-26T15:50:15Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-do-you-override-the-commit-rev-used-by-a-rust-package/47698/3

nyabinary · 2024-08-18T18:00:40Z

We should probably get to this, considering the finalAttrs style is starting to be enforced a lot more.

github-actions bot added 6.topic: rust 6.topic: stdenv Standard environment labels Oct 4, 2022

amesgen force-pushed the buildRustPackage-finalAttrs branch from aa4aa88 to 95a98c9 Compare October 4, 2022 19:53

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Oct 4, 2022

amesgen force-pushed the buildRustPackage-finalAttrs branch from 95a98c9 to a8e675e Compare October 4, 2022 20:20

amesgen marked this pull request as ready for review October 4, 2022 20:37

amesgen requested review from Ericson2314 and matthewbauer as code owners October 4, 2022 20:37

amesgen force-pushed the buildRustPackage-finalAttrs branch from a8e675e to 9e66b79 Compare October 12, 2022 11:47

ghost reviewed Dec 5, 2022

View reviewed changes

amesgen force-pushed the buildRustPackage-finalAttrs branch from 9e66b79 to d16ba39 Compare December 5, 2022 23:58

ghost approved these changes Dec 6, 2022

View reviewed changes

amesgen force-pushed the buildRustPackage-finalAttrs branch from d16ba39 to d28df4f Compare December 27, 2022 13:02

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 and removed 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin labels Dec 27, 2022

SuperSandro2000 reviewed Dec 27, 2022

View reviewed changes

pkgs/build-support/rust/build-rust-package/default.nix Outdated Show resolved Hide resolved

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin and removed 10.rebuild-darwin: 1 10.rebuild-darwin: 1-10 labels Dec 27, 2022

roberth approved these changes Dec 28, 2022

View reviewed changes

amesgen force-pushed the buildRustPackage-finalAttrs branch from d0f22c7 to 172ee4d Compare December 28, 2022 12:52

amesgen force-pushed the buildRustPackage-finalAttrs branch from 172ee4d to fc094d7 Compare January 14, 2023 17:24

amesgen force-pushed the buildRustPackage-finalAttrs branch from fc094d7 to c7be0c0 Compare April 12, 2023 08:38

amesgen requested review from zowoq, winterqt and figsoda as code owners April 12, 2023 08:38

SuperSandro2000 reviewed Apr 12, 2023

View reviewed changes

amesgen force-pushed the buildRustPackage-finalAttrs branch from c7be0c0 to 67d3965 Compare April 12, 2023 13:30

amesgen and others added 3 commits April 12, 2023 15:33

stdenv.mkDerivation: allow to remove "intermediate args"

4e46bb7

This is useful to remove intermediate/temporary arguments used in `finalAttrs`-style derivations.

rustPlatform.buildRustPackage: support finalAttrs style

72d0ef4

Co-authored-by: Sandro <[email protected]> Co-authored-by: Robert Hensing <[email protected]>

difftastic: adopt finalAttrs style

48e9de7

ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 12, 2023

amesgen force-pushed the buildRustPackage-finalAttrs branch from 67d3965 to 48e9de7 Compare April 12, 2023 13:43

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 12, 2023

amesgen commented Apr 12, 2023

View reviewed changes

ShamrockLee mentioned this pull request May 28, 2023

lib.extendMkDerivation, lib.adaptMkDerivation: init #234651

Open

12 tasks

oxalica mentioned this pull request Oct 27, 2023

lanzaboote-tool: init at 0.3.0 #263712

Merged

13 tasks

winterqt mentioned this pull request Feb 13, 2024

buildRustPackage: add overrideRust attribute #288430

Open

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 19, 2024

ShamrockLee mentioned this pull request Mar 20, 2024

darwin: installBinaryPackage init and refactoring of darwin apps #293498

Draft

19 tasks

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 20, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

jian-lin mentioned this pull request Jul 28, 2024

emacs: teach elisp builders the finalAttrs pattern #330589

Merged

13 tasks

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 18, 2024

TomaSajt mentioned this pull request Nov 10, 2024

buildRustPackage: refactor to support finalAttrs pattern #354999

Draft

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustPlatform.buildRustPackage: support `finalAttrs` style #194475

rustPlatform.buildRustPackage: support `finalAttrs` style #194475

amesgen commented Oct 4, 2022 •

edited

Loading

farcaller commented Oct 12, 2022

ghost left a comment

ghost Dec 5, 2022

amesgen Dec 5, 2022

ghost Dec 5, 2022

amesgen Dec 5, 2022

ghost Dec 5, 2022

amesgen Dec 6, 2022

ghost Dec 6, 2022

amesgen Dec 6, 2022

ghost left a comment

nixos-discourse commented Dec 27, 2022

winterqt commented Dec 27, 2022

SuperSandro2000 Dec 27, 2022

amesgen Dec 27, 2022

roberth left a comment

roberth Dec 28, 2022

nixos-discourse commented Jan 14, 2023

jtojnar commented Apr 12, 2023

SuperSandro2000 Apr 12, 2023

roberth Apr 12, 2023

amesgen Apr 12, 2023

roberth Apr 12, 2023

amesgen Apr 12, 2023 •

edited

Loading

9999years commented Feb 13, 2024

amesgen commented Feb 13, 2024

nikstur commented May 14, 2024

nixos-discourse commented Jun 26, 2024

nyabinary commented Aug 18, 2024

rustPlatform.buildRustPackage: support finalAttrs style #194475

Are you sure you want to change the base?

rustPlatform.buildRustPackage: support finalAttrs style #194475

Conversation

amesgen commented Oct 4, 2022 • edited Loading

Description of changes

Things done

farcaller commented Oct 12, 2022

ghost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

nixos-discourse commented Dec 27, 2022

winterqt commented Dec 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roberth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Jan 14, 2023

jtojnar commented Apr 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amesgen Apr 12, 2023 • edited Loading

Choose a reason for hiding this comment

9999years commented Feb 13, 2024

amesgen commented Feb 13, 2024

nikstur commented May 14, 2024

nixos-discourse commented Jun 26, 2024

nyabinary commented Aug 18, 2024

rustPlatform.buildRustPackage: support `finalAttrs` style #194475

rustPlatform.buildRustPackage: support `finalAttrs` style #194475

amesgen commented Oct 4, 2022 •

edited

Loading

amesgen Apr 12, 2023 •

edited

Loading