Package sets within derivations (i.e. python3.pkgs) are not spliced

[rrbutani]

I ran into this while trying to get LLVM to eval under cross-compilation as part of #194634.

Apologies if this has already been addressed elsewhere; I know @Artturin has been working on fixes and cleanup for splicing recently but I didn't see this specific issue brought up in any of the existing issues and PRs.

Problem

As detailed in 888f966, packages that are in package sets within derivations (i.e. the psutil package within python3.pkgs where python3 is a derivation) do not get spliced.

This can be observed via __splicedPackages (I know this is meant to be an implementation detail but it is the attrset that top-level callPackage draws from and it's handy for debugging):

> np = (import <nixpkgs> { }).pkgsCross.aarch64-multiplatform

> np.__splicedPackages.hello ? __spliced
true

> np.__splicedPackages.python3Packages.psutil ? __spliced
true (!!)

> np.__splicedPackages.python3.pkgs.psutil ? __spliced
false

Note that accessing the python package set via python3Packages in the above does give you spliced packages.

---

This happens because of the way splicing operates on derivations:

nixpkgs/pkgs/top-level/splice.nix

Lines 47 to 82 in 6796675

	augmentedValue = defaultValue
	# TODO(@Artturin): remove before release 23.05 and only have __spliced.
	// (lib.optionalAttrs (pkgsBuildHost ? ${name}) { nativeDrv = lib.warn "use ${name}.__spliced.buildHost instead of ${name}.nativeDrv" valueBuildHost; })
	// (lib.optionalAttrs (pkgsHostTarget ? ${name}) { crossDrv = lib.warn "use ${name}.__spliced.hostTarget instead of ${name}.crossDrv" valueHostTarget; })
	// {
	__spliced =
	(lib.optionalAttrs (pkgsBuildBuild ? ${name}) { buildBuild = valueBuildBuild; })
	// (lib.optionalAttrs (pkgsBuildHost ? ${name}) { buildHost = valueBuildHost; })
	// (lib.optionalAttrs (pkgsBuildTarget ? ${name}) { buildTarget = valueBuildTarget; })
	// (lib.optionalAttrs (pkgsHostHost ? ${name}) { hostHost = valueHostHost; })
	// (lib.optionalAttrs (pkgsHostTarget ? ${name}) { hostTarget = valueHostTarget; })
	// (lib.optionalAttrs (pkgsTargetTarget ? ${name}) {
	targetTarget = valueTargetTarget;
	});
	};
	# Get the set of outputs of a derivation. If one derivation fails to
	# evaluate we don't want to diverge the entire splice, so we fall back
	# on {}
	tryGetOutputs = value0:
	let
	inherit (builtins.tryEval value0) success value;
	in
	getOutputs (lib.optionalAttrs success value);
	getOutputs = value: lib.genAttrs
	(value.outputs or (lib.optional (value ? out) "out"))
	(output: value.${output});
	in
	# The derivation along with its outputs, which we recur
	# on to splice them together.
	if lib.isDerivation defaultValue then augmentedValue // spliceReal {
	pkgsBuildBuild = tryGetOutputs valueBuildBuild;
	pkgsBuildHost = tryGetOutputs valueBuildHost;
	pkgsBuildTarget = tryGetOutputs valueBuildTarget;
	pkgsHostHost = tryGetOutputs valueHostHost;
	pkgsHostTarget = getOutputs valueHostTarget;
	pkgsTargetTarget = tryGetOutputs valueTargetTarget;

Only the output attributes of a derivation are spliced; all other attributes are passed through as is.

Attrsets, on the other hand, are spliced recursively (on all attributes):

nixpkgs/pkgs/top-level/splice.nix

Lines 83 to 98 in 6796675

	# Just recur on plain attrsets
	} else if lib.isAttrs defaultValue then
	spliceReal
	{
	pkgsBuildBuild = valueBuildBuild;
	pkgsBuildHost = valueBuildHost;
	pkgsBuildTarget = valueBuildTarget;
	pkgsHostHost = valueHostHost;
	pkgsHostTarget = valueHostTarget;
	pkgsTargetTarget = valueTargetTarget;
	# Don't be fancy about non-derivations. But we could have used used
	# `__functor__` for functions instead.
	} else defaultValue;
	};
	in
	lib.listToAttrs (map merge (lib.attrNames mash));

---

Tying this back to the example above, python3Packages (despite being defined as an alias to python3.pkgs) is spliced because the splice function is passed that attrset when recursively processing the top-level package attrset (whereas when the slice function encounters python3 it does not recurse into python3.pkgs because python3 is a derivation).

Potential Solutions

1: Have pkgs be already-spliced in such places

i.e.: instead of relying on the top-level splicing to splice such package sets, have this be the responsibility of the package set's scope.

Tangent: Splicing and Scopes

There's already some precedent for this; today scopes that use makeScopeWithSplicing are automatically given a newScope and a callPackage that contains spliced versions of the packages in the scope:

nixpkgs/lib/customisation.nix

Lines 280 to 305 in 6796675

	makeScopeWithSplicing = splicePackages: newScope: otherSplices: keep: extra: f:
	let
	spliced0 = splicePackages {
	pkgsBuildBuild = otherSplices.selfBuildBuild;
	pkgsBuildHost = otherSplices.selfBuildHost;
	pkgsBuildTarget = otherSplices.selfBuildTarget;
	pkgsHostHost = otherSplices.selfHostHost;
	pkgsHostTarget = self; # Not `otherSplices.selfHostTarget`;
	pkgsTargetTarget = otherSplices.selfTargetTarget;
	};
	spliced = extra spliced0 // spliced0 // keep self;
	self = f self // {
	newScope = scope: newScope (spliced // scope);
	callPackage = newScope spliced; # == self.newScope {};
	# N.B. the other stages of the package set spliced in are *not*
	# overridden.
	overrideScope = g: makeScopeWithSplicing
	splicePackages
	newScope
	otherSplices
	keep
	extra
	(lib.fixedPoints.extends g f);
	packages = f;
	};
	in self;

Here's how the python scope is set up, for example:

• the set of python overlays and the python package sets from the other pkgsBuildBuild, pkgsBuildHost, etc. sets are given to makeScopeWithSplicing; this yields the pythonPackage fixed point:
  

  nixpkgs/pkgs/development/interpreters/python/default.nix

  Lines 43 to 77 in 7f6ecd4

  	# Function that when called
  	# - imports python-packages.nix
  	# - adds spliced package sets to the package set
  	# - applies overrides from `packageOverrides` and `pythonPackagesOverlays`.
  	({ pkgs, stdenv, python, overrides }: let
  	pythonPackagesFun = import ./python-packages-base.nix {
  	inherit stdenv pkgs lib;
  	python = self;
  	};
  	otherSplices = {
  	selfBuildBuild = pythonOnBuildForBuild.pkgs;
  	selfBuildHost = pythonOnBuildForHost.pkgs;
  	selfBuildTarget = pythonOnBuildForTarget.pkgs;
  	selfHostHost = pythonOnHostForHost.pkgs;
  	selfTargetTarget = pythonOnTargetForTarget.pkgs or {}; # There is no Python TargetTarget.
  	};
  	hooks = import ./hooks/default.nix;
  	keep = lib.extends hooks pythonPackagesFun;
  	extra = _: {};
  	optionalExtensions = cond: as: if cond then as else [];
  	pythonExtension = import ../../../top-level/python-packages.nix;
  	python2Extension = import ../../../top-level/python2-packages.nix;
  	extensions = lib.composeManyExtensions ([
  	pythonExtension
  	] ++ (optionalExtensions (!self.isPy3k) [
  	python2Extension
  	]) ++ pythonPackagesExtensions ++ [
  	overrides
  	]);
  	aliases = self: super: lib.optionalAttrs config.allowAliases (import ../../../top-level/python-aliases.nix lib self super);
  	in makeScopeWithSplicing
  	otherSplices
  	keep
  	extra
  	(lib.extends (lib.composeExtensions aliases extensions) keep))

• that's then exported as a passthru on the python3 derivation as pkgs:
  

  nixpkgs/pkgs/development/interpreters/python/default.nix

  Line 97 in 7f6ecd4

  pkgs = pythonPackages;

Crucially, the spliced packages makeScopeWithSplicing produces are not made available in the actual scope (i.e. the package set; python3.pkgs in the above) itself but are made available to packages within the scope via the callPackage machinery.

This actually means that the contents of scopes like python3Packages (i.e. scopes that are exposed directly – not via a derivation – and are spliced at the top-level) are actually spliced "twice": once as part of the makeScopeWithSplicing call (accessed by members of the scope) and then again at the top-level (accessed by members of the outermost scope, the top-level).

It would be nice if we could reuse the splicing.

Using the outer scope's splicing within the inner scope is easy to do:

• we can:
  • pass in the path to the scope being constructed + __splicedPackages to this function
  • and then use __splicedPackages. ... in lieu of spliced0

But this is problematic in cases where the scope is overriden (i.e. overrideScope) outside of an overlay-like context (where the top-level binding of that scope is also updated). In such cases, the scope's callPackage will continue using the original pre-overrideScope version of the packages within the scope because the splicing would still be pulling things from the top-level __splicedPackages. Put another way: overlays = [(f: p: { python3Packages = p.python3Packages.overrideScope (_: _: { ... }); })] would be okay because nixpkgs.python3Packages is updated to point to the overriden scope but just doing (nixpkgs.python3Packages.overrideScope (_: _: { ... })).some-package would not be okay.

Going the other way (using the scope's splicing for the top-level) seems a little trickier but wouldn't have this issue. We'd need to have the makeScopeWithSplicing expose the spliced attrset with something like a __splicedPackages attr and we'd then want spliceReal's handling of attrset to check for such an attr and use it instead of redoing the splicing itself. I don't think this runs into any recursion issues but I have not tested this yet.

Ultimately this (splicing scopes multiple times) is somewhat orthogonal to this issue but the above has some overlap with the potential solutions below and might influence a decision there.

---

Now that we know how scopes get spliced:

The most straight-forward way to have our scope yield an already-spliced pkgs attr is probably to get makeScopeWithSpliced to give us it's spliced attrset. We can modify makeScopeWithSpliced to expose the attrset as __splicedPackages = spliced (as discussed above) and then swap out this line:

nixpkgs/pkgs/development/interpreters/python/default.nix

Line 97 in 7f6ecd4

pkgs = pythonPackages;

for pkgs = pythonPackages.__splicedPackages.

We would also have to replicate this change for all other users of makeScopeWithSpliced that export their package set as part of a derivation's attrs.

2: Adjust spliceReal's handling of derivations

Recursing on all of the attributes of every derivation seems fraught but maybe it's safe to recurse on drv.passthru or an opt-in list of attributes (i.e. we could have spliceReal look for a passthru attr named __spliceRecurseAttrs on derivations) or maybe even just pkgs (since that seems to be the convention used).

This has the benefit of not requiring any changes from users of makeScopeWithSpliced and handling splicing for packages that are referenced via a derivation's attrs (depending on how general we adjust spliceReal to be on derivation attrs).

3: Discourage using package sets like python3.pkgs "directly"

(and instead push people to use python3Packages, lua5Packages, etc. in nixpkgs)

This seems suboptimal, both because this will be another thing that'd need to be enforced in nixpkgs to have cross work for packages and because the foo.pkgs.bar pattern (where foo is a derivation) seems pretty pervasive in nixpkgs (python, lua, perl, postgresql, etc.).

---

A version of option 2 (with the __splicedPackages attr for deduplicating the work of splicing scopes as a follow-up PR if it doesn't cause breakage) seems like the least-worst fix to me but I'm not particularly satisfied with any of these solutions; hopefully there's a more elegant solution that I'm missing 🤞.

cc: @Artturin

[ghost]

Potential Solutions

4. Reconsider whether or not splicing was a good idea

There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.

[Artturin]

Potential Solutions

4. Reconsider whether or not splicing was a good idea

There has got to be a simpler way of doing all of this. The fact that splicing breaks randomly in weird ways and hardly anybody understands it is a major contributor to people hating on cross compilation. There has got to be a simpler way to do all of this.

#204303 (comment)

[FRidh]

3: Discourage using package sets like python3.pkgs "directly"

I think about 8 years ago I introduced this and it has been a clear mistake. Since then this pattern was copied throughout. We should indeed go to pythonPackages and also move the helpers such as buildEnv and withPackages into the package set, like haskell already had before.

[AndersonTorres]

@FRidh what are the costs of reverting it?

[FRidh]

@FRidh what are the costs of reverting it?

• moving all the helpers (then we can also start using .overrideScope)
• adjusting to python3Packages throughout
• keeping the old way of working in place for a release or 2

Basically people willing to put in that effort.

[AndersonTorres]

"Willing" also includes having some theoretical knowledge?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/1