Standardize all-packages.nix arguments and stdenv attributes

[Mathnerd314]

There's a lot of duplication between the cross-building stuff for gcc and the Linux stdenv bootstrap.
Similarly, in libiconv's setup, the same logic is repeated twice, once for cross-building and once for stdenv.
Also, consider the dropbox package, which checks stdenv.system to see which binary to download. Supposing that one was cross-compiling a NixOS system from OS X, this is incorrect; stdenv.system would be x86_64-darwin, while the desired value x86_64-linux would not even be available in the current crossSystem attributes. (The closest is config, which would be "x86_64-unknown-linux-gnu"). On the other hand, the url contains the arch, which is present in crossSystem, but not in stdenv.

In other words, it's unclear what attributes are supposed to be checked for platform-dependent behavior, and the set of attributes available is inconsistent between cross-builds and normal builds. This should be decided upon and the results documented somewhere (e.g. nixpkgs manual).

Also see #4855 for a related discussion on cross-building dependencies, #1928 for requested stdenv attributes and cross-building TODO's that never happened, #6221 and #7334 for more stdenv attributes, #8081 for bootstrap tools scope creep, #3574 and #7384 mentioning dedicated Hydra job for bootstrap tools, #6952 for a possible alternative to our current bootstrapping approach, #5944 for why cross-compiling only works with gcc, #4963 for ideas on NixOS (as opposed to nixpkgs) cross-compiling, #2817 for odd stdenv overrides behavior, and #4998 for the likely end state of this issue.

I propose setting system to be the crossSystem-style attribute set for the targeted system, and combining bootStdenv and crossSystem into bootSystem, representing the host system on which the builders are run.

[ghost]

Today, by coincidence, I tried cross-compiling things with mingw but that failed because of libiconv, so it might not only be duplicated but more importantly, it does not work. libiconv error was caused by assert !stdenv.isLinux or something like that, which should be an assertion on the target and not on the host. Thanks a lot for looking into this!

[vcunat]

I personally believe the standard attributes (buildInputs, stdenv.system, etc.) should be for the target platform where the output is to be executed, as that seems to be the most common use case. (Note that I've never done any cross building, though :)

[Ericson2314]

I've recently been mulling this. We have all this .nativeDrv and .crossDrv and native vs normal dependencies, so that there is just one package set when cross compiling. But it might just be simpler to have separate build- and target- package sets, and just have them coincide in the non-cross-compiling case.

[vcunat]

We could have separate sets, but how would that go with passing dependencies into packages? Now we only put the attribute name in there, without specifying whether it's a native or cross package...

[Ericson2314]

@vcunat it would be a huge breaking change on the callPackage ideom, I'm afraid. I'd hope to stage it together when we overhaul the ideom to better expose flags like with the nixos module system.

[Ericson2314]

Well, then I think both ways could be done concurrently at the cost of even more complexity in the interim. stdenvCross could expose a special callPackage instead of mkDerivation that would pass in both the native and cross package using __toString__ to default to the cross one.

[Ericson2314]

Basically, I see why the current approach was chosen to minimize churn, but long term I'd like to do it what I see is the "right way".

[Ericson2314]

@Mathnerd314 Did the all-packages fix-point cleanup affect this at all? Notwithstanding what I wrote above, everything you proposed sounds like a good idea and I'd like to see it again.

Just now I was yet again confronted with .nativeDrvs being purposely different, and that lead me back here.

[vcunat]

Did the all-packages fix-point cleanup affect this at all?

I don't think so. This is about meaning of system, isLinux, etc.

[Ericson2314]

@vcunat sure, not saying that PR actually changed semantics. But curious whether or not it made anything easier to implement.

[vcunat]

The difficulty here won't be in implementation, I think. We probably want to break backwards compatibility here (change meaning), which isn't an easy thing to do and some may oppose it. Also, we should design such changes carefully to avoid the need to change these soon again.

[Ericson2314]

@vcunat thanks for the clarification--sounds good to me. Time to get designing then!

I propose setting system to be the crossSystem-style attribute set for the targeted system, and combining bootStdenv and crossSystem into bootSystem, representing the host system on which the builders are run.

Hmm, I thought crossSystem was the targeted system, and the problem is we don't have a similarly-detailed way to talk about the host system.

[vcunat]

crossSystem.config is what is passed to --host (with autotools stuff), but that binding won't exist when not cross-compiling. (Then crossSystem == null and that's a common test, too.)

[Ericson2314]

Ah yes. Whereas I'd have thought bootStdenv had something to do with autotool's "build" (not "host" or "target");

[Ericson2314]

Ok so thoughts so far:

• I don't know what bootStdenv is, but if it should go away. less concepts is probably better here. From its 3 occurrences in all-packages.nix, it is just being propagated back to itself, so it's almost ready to be purged from there.
• There should be a hostSystem and targetSystem flags. hostSystem is basically crossSystem when cross-compiling, targetSystem is a current (IMO mis)use of crossSystem where we use it to configure the tools which will be used in the cross stdenv (gccCross, binutilsCross).
  • Besides the fact that it better conveys intent, I want this so it's possible to get a set of native pkgs configured to build for the current target, without setting up a cross-compiling stdenv / cross-compiled pkg set.
• I think my idea from before can be bailed out:
  • nativeDrv and crossDrv is always a bad idea because it implies that the set of packages is the same on both platforms, and it is too easy to forget which to use with e.g. ${myPkg} in a string. But for the time being we need to keep it. At least, there should be a warning when it falls back on the default because one isn't chosen.
  • (propigated)(native)buildInputs (modulo case) can all be deprecated, instead there can be just deps and propagatedDeps. These two new ones will assert that anything given them does not have any crossDrv nativeDrv attrs (i.e. the right one has been selected).
  • nativeDrv can just select the package from the targetSystem != null native package set I described earlier. Actually, it should assert that this is the same as today's nativeDrv while we transition, as I believe this will allow us to catch tons of broken packages.
  • There can be a new callPackageDuo where one gives a two arg function taking separate a set of native and cross deps. This exists side by side with the old callPackage so their is no backwards compat issue.
• Don't conflate dependencies in setup.sh #4855 offsers some good inspiration:
  • Packages can issue a warning if they are used in an unintended way (cross deps on build tools, native deps on libraries). since nativeDrv is taken from the targetSystem != null package set, we won't trip the warning (or rather the assert I proposed would trip it, but maybe it can squelch the output).
  • There can be a callPackageNg which given a single set of packages, selects the default variant (same mechanism as warning). Note this naturally goes well with deps and propagatedDeps too.

[Ericson2314]

To be clear, top-level/all-packages.nix would take hostSystem and targetSystem (and maybe a buildSystem too). top-level/default.nix however could take some sort of list list, considering how package sets are chained for cross-compiling:

(columns share value):
build  - host   - target
         build  - host   - target
                  build  - host   - target
                           build  - host   - target
                                    ...

Staring at the picture, the list of platforms must be at least one platform long---nulls are appended so the length of the list is >=3. Additional nulls can be used in the list to trigger a "self-bootstrap" at any point.

For example: [ <x86-linux> <x86-solaris> null <arm-linux> null null null ] yields:

 <x86-linux> <x86-solaris> null
             <x86-solaris> null          <arm-linux>
                           <x86-solaris> <arm-linux> null
                                         <arm-linux> null        null
                                                     <arm-linux> null null

Translated:

1. Build from x86 linux, packages to run on x86 solaris, targeting x86 solaris.
2. Build from x86 solaris, packages for x86 solaris, targeting arm linux. Because of that first null, any tools will be built on solaris, rather than linux.
3. Build from x86 solaris, packages for arm linux, targeting arm linux.
4. Build from arm linux, packages for arm linux, targeting arm linux.
5. Again, build from arm linux, packages for arm linux, targeting arm linux. With intensional store, this should be a no-op after building tool chain on arm if it is bit-wise identical.

Note that because null means "same as before" the list can be thought of as some sort of derivative-like thing.

Now obviously this is way overkill for practical uses, but I believe supporting arbitrary lists of this sort will be a good way to battle-harden the cross-compiling infrastructure and ensure we implemented it correctly.

[Ericson2314]

Looking at https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/default.nix#L8-L33, what is the relationship between system and platform (I gather these would be bundled into buildSystem if we go that route)?

[Mathnerd314]

(propogated)(native)buildInputs (modulo case) can all be deprecated, instead there can be just deps and propagatedDeps.

This is similar to bjornfor's approach (#4855 (comment)). It sounds fine to me; I suggest discussing it there so it can get consensus. This issue is not about nativeDrv or crossDrv or anything like that (although stdenvCross is slightly relevant).

get a set of native pkgs configured to build for the current target, without setting up a cross-compiling stdenv / cross-compiled pkg set.

My idea is that bootstrapping and cross-compiling are very similar. There's no way to build "native" packages without a bootstrap compiler, and similarly there's no way to build cross-packages without a cross-compiler. With the right design, they can be passed around using the same code, so that "native" builds are cross-builds where the host/target are the same and the host tools are the bootstrap tools.

build - host - target

Target is a GCC-ism; clang always builds with all targets available. And GCC can be built and run twice (and actually is in the nixpkgs bootstrap); there is no need to support the "Canadian Cross", only one level of build/host or host/target distinction is needed.

In my proposal it is bootSystem attribute / normal system attribute, I guess you could keep bikeshedding this.

chaining cross-compiles

This could be jury-rigged if needed by providing a "builtSystem" attribute in the package set such that it could be used for "bootSystem", so you would do

let pkgsFun = system: bootSystem: import all-packages.nix { inherit system bootSystem; };
in pkgsFun "arm" (pkgsFun "x86" initialBootSystem).builtSystem

But nixpkgs would not need to support more than one level of cross-compilation at a time.

I don't know what bootStdenv is

It's a parameter used in most of the stdenv's to build gcc and other base tools; the final stdenv consists of tools built with bootStdenv=stage4, and similarly stage4 was built with bootStdenv=stage3, down to the bootstrap tarball which has a null bootStdenv.

what is the relationship between system and platform

platform is an attribute set with a few gcc build options (fpu, cpu, etc.) and some kernel config.
system is a string like "x86-linux", it's passed through to mkDerivation to ensure that Hydra uses the right system to compile each derivation.
crossSystem is the interesting attribute set, it contains a "libc" attribute which actually does something and then some other mostly-unused things like a second platform attribute set.

We probably want to break backwards compatibility here (change meaning)

I consider "backwards-compatibility" to mean that nixpkgs / nixos configurations don't need to change, and I think that is the case here. platform / system / crossSystem are internal to nixpkgs.

[Ericson2314]

#4855 (comment)

Yeah I was inspired by that. A slight difference is I think the intended use of each package should only be a default---any package should be able to be used as a build or host dep with enough explicitness/verbosity.

My idea is that bootstrapping and cross-compiling are very similar. There's no way to build "native" packages without a bootstrap compiler, and similarly there's no way to build cross-packages without a cross-compiler. ...

Definitely! Right now "stdenv" is a nebulous/overloaded concept---a set of default packages for mkDerivation and a mini-package set for bootstrapping purposes. Now that all-packages is no longer insanely fixed, I think its just simpler to just use entire pkgsets both when booting/cross-compiling---turn bootStdenv into bootPkgs. And it sounds like that's what you want too?

Target is a GCC-ism; clang always builds with all targets available.

I'm definitely sympathetic---I've always been shocked and appalled not everyone does it the LLVM way, building for all targets. But that said, while software exists that can't do multi-target, we're kind of bound to support the concept. And LLVM-based tool chains in practice seem to still use binutils not lld, so nobody is completely avoiding this in practice.

Platform vs System.

Thanks for explaining it---reminds me a lot of https://github.com/rust-lang/rfcs/blob/master/text/0131-target-specification.md. It seems really we ought to derive system from platform, and the system string really ought to be a LLVM's target triple if it exists from all) but meaningful changes here might depend on changing nix/hydra?

And GCC can be built and run twice (and actually is in the nixpkgs bootstrap);

Hmm not sure what you mean? Certainly bootstrapping involves building multiple GCCs no doubt about it.

There is no need to support the "Canadian Cross", only one level of build/host or host/target distinction is needed. In my proposal it is bootSystem attribute / normal system attribute, I guess you could keep bikeshedding this.

This could be jury-rigged if needed...

So I guess with a bootPkgs setup, technically only the host and target configurations need to be specified. The build configuration can be gotten from bootPkgs's "host" config, (and bootPkgs's target configuration asserted to match the being-constructed pkg set's host config). Of the host and target configs, the latter is far more detailed---things like linux header version don't matter as building a compiler supporting x target should not mean building libraries, downloading src, etc for that target.

[Mathnerd314]

"stdenv" is a nebulous/overloaded concept

I think of stdenv as something similar to Debian's "build-essential", containing all packages necessary for building simple C programs.

a mini-package set for bootstrapping purposes

IMO all packages should be able to bootstrap themselves, we are pretty close to that (fetchurl is now built-in to Nix)

use entire pkgsets when booting/cross-compiling---turn bootStdenv into bootPkgs

bootSystem could have a pkgs attribute. I'm not sure how that would work exactly.

GCC can't do multi-target, so we're bound to support the concept

GCC can be built and run twice (and actually is in the nixpkgs bootstrap);
Hmm not sure what you mean?

What I meant was that, for GCC, using only compilers with host==target (native compiler) or build==host (cross compiler) is sufficient.

If you wanted to do something with build != host != target, then you would build a native compiler for the host and a cross-compiler for the target (on that host). It's an extra GCC build but pretty small in the grand scheme of things. And regardless of Nixpkgs support, using a build!=host!=target setup in Nix would require a build-farm so that it could build on multiple platforms.

It seems really we ought to derive system from platform

I think platform should contain the system attribute (and remove it from all-packages arguments), the rest is academic.

[Ericson2314]

bootSystem could have a pkgs attribute. I'm not sure how that would work exactly.

Still trying to clarify my thoughts on the other stuff, but let me start here. This should be nested the other way around — the concept of a boot system makes no sense without packages for the boot system. Nix can only run code on a given platform if pacakges (or "imaginary" impure packages) exist for that platform.

[vcunat]

I think it would be a mistake to try to fix this issue perfectly in a single iteration. (holds more generally)

Let's make a substantial improving step, e.g. excluding support for the canadian cross etc for now. We'll see after some time how it works out and we might reiterate. (Or not, if the issue won't be pressing enough anymore, but both are positive cases.)

[Ericson2314]

@vcunat Well I didn't say one big PR. I think I've been here long enough to know how that will go :). But I'd like to have some sort of long term plan, just because its all pretty non-trivial.

I think of stdenv as something similar to Debian's "build-essential", containing all packages necessary for building simple C programs.

I want it to just be that, but while we bootstrap stdenvs specifically its something more.

IMO all packages should be able to bootstrap themselves, we are pretty close to that (fetchurl is now built-in to Nix)

I mean stdenv as a package set is bootstrapped, i.e. all the stageN are stdenvs, not arbitrary package sets.

What I meant was that, for GCC, using only compilers with host==target (native compiler) or build==host (cross compiler) is sufficient.

If you wanted to do something with build != host != target, then you would build a native compiler for the host and a cross-compiler for the target (on that host). It's an extra GCC build but pretty small in the grand scheme of things. And regardless of Nixpkgs support, using a build!=host!=target setup in Nix would require a build-farm so that it could build on multiple platforms.

I thought about this for a long, long time, and now agree, though perhaps for different reasons. build != host != target compilers not only are strange, but actually useless for us. Consider that with chains of package sets as I defined them above, the dependency dag for any package can't be longer than than the number of package sets (a build dep of a build dep comes from the build pkg set's pkg set, yuck). Worse of all, propagated build deps are now completely broken as they may not run on the same platform.

Instead every "native round" we need to fix the package set so that it's build dep is itself. Unlike our fix-extend ideom here, the idea is that if a package is "already" defined, we don't override it. this ensures bootstrapped packages aren't infinitely bootstrapped. The three forms are roughly

let
  stage0 = ... bootstrapTools ...;
  fixedStageSuccN = mkPkgs { bootPkgs = fixedStageSuccN ; ... } // StageN; # "final native stages"
  overridingStageSuccN = mkPkgs { bootPkgs = StageN; ... }; # today's stdenv stages or cross stages.

Now I feel that perhaps mkPkgs should still take separate buildPkgs, hostPlatform and targetPlatform parameters, and just assert:

(buildPkgs.targetPlatform == hostPlatform) # Basic sanity
  && (buildPkgs.buildPkgs.hostPlatform == buildPkgs.hostPlatform ) # Inductively ensures closure property, I hope
  && (buildPkgs.hostPlatform == hostPlatform) || (targetPlatform == hostPlatform) # No Canadian cross

just because anything that cares about targetPlatform probably uses autotools anyways. This is a minor quibble, however.

[Ericson2314]

Ok, to actually get to specifics, here is what I think the first step should be.

Currently top-level/* calls into stdenv/, and also vice versa. I think we can and should break the cycle, and the correct and easier, but perhaps counter-intuitive way to do this, is only have stdenv/* call top-level/*. What do you all think?

[Ericson2314]

A Few notes:

• Impure defaults https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/default.nix#L9 and https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/default.nix#L42-L89 can be moved to some other file, they are passed to whatever the entry point in stdenv/* turns out to be.
• https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/stdenv.nix probably belongs elsewhere I think.
• https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/default.nix#L138-L146 stdenv stages would call this function and perhaps provide the chain of things to be fixed.

[vcunat]

See closely related issue #14965

[Ericson2314]

NB I've just started prototyping these things in https://github.com/Ericson2314/nixpkgs/tree/cross

[vcunat]

Today I started prototyping stuff around #14965, but it seems we won't clash due to focus on different aspects of cross-compilation.

[Ericson2314]

@vcunat do you have branch / want to collaborate on this?

[Ericson2314]

Ok, actually have some meaningful work there now. Calling it a night after getting seemingly-undebuggable stack overflow when evaluating :/.

[bennofs]

@Ericson2314 is this still an issue?

[Ericson2314]

We're getting close! But yeah there's bits of this that are still not done. if somebody wants to edit the OP to add a giant checklist of other issues for each of @Mathnerd314's excellent points, that would be great.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[siraben]

@Ericson2314 what's the status of this issue?

[Mathnerd314]

As author of this issue, this looks like it's under control now. There's a single set of platform-specific attributes defined via https://github.com/NixOS/nixpkgs/tree/master/lib/systems and mostly everything uses those. Furthermore the arguments localSystem / crossSystem to all-packages.nix are exactly the platforms exposed in stdenv.hostPlatform / stdenv.buildPlatform / stdenv.targetPlatform. So the original issue of system overlapping with platform and things being scattered in stdenv is gone. Looking through the discussion I don't see anything that hasn't been resolved, so I'd say to close it, but I'll wait for @Ericson2314 to chime in as he did all of the work.

[Ericson2314]

Wow, I looked over everything in this thread, and a few of the linked issues and I think it's all done!* This is the issue that started it all for me more than any other, so It's a nice moment.

*I'm still peeved that import <nixpkgs> takes all three of system platform and localSystem with no deprecation in site, but that post-dates this issue even if it's one of the last things in the same spirit.