Package set standardization

[tomberek]

Package Set Standardization

Ensure the package sets in Nixpkgs use a common interface and implementation.

Problem

Package sets are a concept used in Nixpkgs, but not often documented and explained. This is the mechanism that is used to create a scope (https://github.com/NixOS/nixpkgs/blob/master/lib/customisation.nix) and supports the callPackage design pattern. There are a few of these in Nixpkgs, top-level, pythonPackages, perlPackages, etc. Any attempt to manipulate these in a common manner runs into issue where they concept is used/implemented/exposed in a slightly different way. Eg perlPackages and phpPackages are constructed differently:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/php-packages.nix
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/perl-packages.nix
Another place of complexity is pythonPackages.

Many of the package sets do seem to follow a common pattern:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/octave-packages.nix#L24
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/qt6-packages.nix#L13

Consequences

The mechanism is often discussed and confused with overlays.
The mechanism can be confusing, but also helpful: https://summer.nixos.org/blog/callpackage-a-tool-for-the-lazy/

Tooling to manage and manipulate these package sets become more awkward.

Potential issues

Cross-compilation and makeScopeWithSplicing are more complex: https://github.com/NixOS/nixpkgs/blob/master/lib/customisation.nix#L282
Picking an implementation that can accommodate the various needs. (what are they?)
Duplicate of https://github.com/nixpkgs-architecture/issues/issues/1 ?

[infinisil]

I recently had the chance to think about how I'd design a new package set: #235614. Afaik the only thing missing is cross compilation support.

I think we can treat this as separate from #295046 because this can be much smaller in scope. We can still have various override mechanism, but at least they'd be consistent between package sets.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-08-nixpkgs-architecture-team-meeting-42/31454/1

[Artturin]

x.pkgs is bad because package sets within derivations are not spliced
#211340

#211340 (comment)

I think about 8 years ago I introduced this and it has been a clear mistake. Since then this pattern was copied throughout. We should indeed go to pythonPackages and also move the helpers such as buildEnv and withPackages into the package set

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-28-nixpkgs-architecture-team-meeting-46/36171/1