Clarify in documentation which configurations of host, exec and target are supported.

[uri-canva]

@aherrmann in #209 (comment) you clarify that two major use cases are supported by rules_nixpkgs:

1. Nix provides Bazel and perhaps other development tools AND rules_nixpkgs imports system dependencies from Nix into Bazel.
2. Vanilla Bazel AND "bindist" or Bazel built system dependencies. I.e. no Nix involved at all.

Translating these into host, exec and target it implies that rules_nixpkgs supports:

1. host = nix, exec = nix, target = nix
2. host = non-nix, exec = non-nix, target = non-nix

It does not support any other combinations. I think this limits the value of rules_nixpkgs a lot: if host, exec, and target are all nix, then one might want to use buildBazelPackage from nixpkgs. If host, exec, and target are all non-nix, then one does not need to use rules_nixpkgs at all. With this narrow focusrules_nixpkgs seems to fit into a very niche usecase of a nix centric development model that needs to support windows, so that buildBazelPackage becomes tricky to keep in sync with the windows specific bazel setup.

In practice I don't know how many people are using rules_nixpkgs like that. Personally I can say the way we've been using it fits into:

3. host = nix, exec = nix, target = non-nix
4. host = nix, exec = non-nix, target = non-nix

Given https://github.com/digital-asset/daml/blob/9cbe57b073444a0087e3bc8cc26c57cacd037257/bazel_tools/packaging/package-app.sh, I believe the use case for that repo is 3 as well, since if the target was a nix system, patching the binaries to remove references to the nix store would not be necessary.

However this use case has been hitting a lot of challenges both in rules_nixpkgs and upstream nixpkgs. For example upstream bazel is built in a way that target must be a nix system, since it hardcodes some nix store paths in shebangs in parts of the target code (for example the python stub).

Discussion in #180 highlights the challenges of not clarifying the "nixness" of a particular platform, specifically the exec one. The suggestions are wildly different depending on whether we want exec to be nix or not: when not using remote execution, then 3 is a very common setup, since we have nix on the host anyway, and the exec is the host. However once you put remote execution in the mix, it's much more likely that you'll end up wanting 4 instead.

Note that I've been referring to host, exec and target since that's what bazel calls them. In nix they would map to build, host and target respectively. See https://nixos.org/manual/nixpkgs/unstable/#chap-cross for more details. nixpkgs does not make which platforms are nix and which are non-nix very clear, but it does have at least some very clear non-nix platforms, like ios and android.

[uri-canva]

Let me clarify that my personal opinion is that we should support 4 exclusively because:

1. is supported by buildBazelPackage in nixpkgs, and the general nix build tooling / setup
2. is supported by upstream bazel without using rules_nixpkgs
3. makes it very difficult to:
   1. support remote execution
   2. use rulesets that rules_nixpkgs doesn't support / doesn't have a nixpkgs_*_configure rule for
   3. use prebuilt native dependencies from nix agnostic third party package repositories such as maven, pypi during the build

[uri-canva]

My ideal use of bazel + nix + rules_nixpkgs would be to rely on bazel's own mechanisms for reproducibility and hermeticism as much as possible, and only use nix for the parts that bazel relies on the environment for. For example I'd be happy to use go_sdk from non-nix prebuilt binaries, but I'd still want to use nixpkgs_cc_configure rather that configuring local_config_cc based on the PATH.

[uri-canva]

Also maybe more controversial I'd like rules_nixpkgs to work with both bazel from nix and bazel from upstream, since bazel from bazelisk with embedded jdk could be hermetic enough.

[aherrmann]

Thank you for raising this issue @uri-canva, it’s an important topic to clarify!

Translating these into host, exec and target it implies that rules_nixpkgs supports:

1. host = nix, exec = nix, target = nix
2. host = non-nix, exec = non-nix, target = non-nix

The linked comment considers two axes:

• Whether Bazel is provided by Nix, or not (i.e. a vanilla binary distribution).
• Whether system dependencies (toolchains/system libraries) are provided by Nix through rules_nixpkgs; or not (i.e. without rules_nixpkgs involved).

These do not mention “host”, “exec”, “target” platforms, and there are other supported cases than the two cases listed in your comment above.

In particular, rules_nixpkgs already supports both, use-cases where “target = nix”, and where “target = non-nix". See for example this blog post on fully statically linked Haskell binaries or this WIP PR for Skyscope. See also #291 which aims to better document deployment models such as the one from Daml that you link. This is also reflected in the target constraints that rules_nixpkgs provided toolchains define.

About the case where “exec = non-nix”. I think this reflects a very specific use-case that requires significant divergence from standard nixpkgs. Most Nix derivations will have run-time dependencies into the Nix store. This includes compiler toolchains. Runtime dependencies of compilers translate to “exec” platform constraints in Bazel. So, to support this use-case (as I understand it) you will require a lot of custom overrides and patches to avoid such Nix store path runtime dependencies. To be clear, I’m not saying you shouldn’t do this. But, I don’t think we can force all rules_nixpkgs users to carry that burden.

Either way, I would like to better understand in which way rules_nixpkgs is currently preventing this use-case. One issue I see is that the “support_nix” exec constraint is currently hard-coded. I would not be opposed to making that configurable. However, we cannot remove it altogether, as explained here. The other issue I see is that rules_nixpkgs creates symlinks into the Nix store. Are there any other issues you’re facing with this approach?

Let me clarify that my personal opinion is that we should support 4 exclusively because:

Some comments to your points 1, 2, and 3:

1. buildBazelPackage is about building a Bazel package with Nix, typically as part of nixpkgs. The entry-point to the user is Nix. This is not incremental w.r.t. the Bazel targets in the package: Any change to the code or a dependency will trigger a full, uncached rebuild of the Bazel project. rules_nixpkgs on the other hand is about providing fine-grained access to Nix provided toolchains and other dependencies to a Bazel project. The entry-point to the user is Bazel (modulo Nix shell). So, I'd view those as separate use-cases and would not view buildBazelPackage as a replacement for rules_nixpkgs.
2. To clarify, rules_nixpkgs ”support” of this use-case only extends so far as to get out of the way in this case. E.g. to support a Unix + Windows cross-platform project.
3. On i., yes, we’re working on it. On ii., can you clarify in which way this fails for you? On iii. could you also clarify? Usually NixOS is the problem here, not rules_nixpkgs.

For example I'd be happy to use go_sdk from non-nix prebuilt binaries, but I'd still want to use nixpkgs_cc_configure rather that configuring local_config_cc based on the PATH.

In which way does this fail for you? I would expect an incompatibility between the Nix provided C toolchain and the downloaded Go toolchain. But, that falls outside of rules_nixpkgs.

[uri-canva]

These do not mention “host”, “exec”, “target” platforms

I know, but I still believe it maps to those:

Whether Bazel is provided by Nix, or not (i.e. a vanilla binary distribution).

Bazel from nixpkgs can only target nix systems, even when built with enableNixHacks = false (which is the default). The rewritten shebangs in all the scripts that end up in the bazel outputs all refer to nix derivation outputs, I mentioned the python stub, which is configurable, but the same happens for the java binary wrapper and many other pieces. This is why I think using bazel from nixpkgs with rules_nixpkgs implies the target platform is nix. It also implies that host is a nix system, since bazel from nixpkgs can only run on a nix system.

Whether system dependencies (toolchains/system libraries) are provided by Nix through rules_nixpkgs; or not (i.e. without rules_nixpkgs involved).

I think the distinction is whether the toolchains/system libraries are provided by nix, regardless of rules_nixpkgs. For example using nix direnv to bring a c toolchain into the PATH and letting bazel autodetect it would be a way of having nix toolchains without rules_nixpkgs. The comment I linked to puts it like this:

"bindist" or Bazel built system dependencies. I.e. no Nix involved at all.

This case to me implies exec has to be nix, as the nix binaries in the toolchains can only run on a nix system, and to a lesser extent target has to be nix, because the output of some of those binaries or outputs built using those dependencies can only run on a nix system, for example because binaries link against nix libraries.

That's why I mapped the set of two supported use cases:

1. Vanilla bazel with toolchains and dependencies for vanilla bazel
2. Nixpkgs bazel with nixpkgs toolchains and dependencies

to the two combinations of:

1. host = nix, exec = nix, target = nix
2. host = non-nix, exec = non-nix, target = non-nix

there are other supported cases than the two cases listed in your comment above.

That's true, while the two supported use cases in terms whether bazel is vanilla or nixpkgs, and whether toolchains and dependencies are vanilla or nixpkgs, imply that at least those two combinations are supported, it doesn't not necessarily imply that at most those two combinations are supported. However here a distinction must be drawn on what it means for something to be supported by rules_nixpkgs:

• it is technically possible for a user to implement, and rules_nixpkgs does not present an impediment to it.
• rules_nixpkgs provides the mechanisms for the user to implement it.

I'm much more interested in the second one: after all the purpose of a dependency is to provide a capability in a way that doesn't require the dependant to implement it, but to prevent misunderstandings we should be explicit about the distinction.

In particular, rules_nixpkgs already supports both, use-cases where “target = nix”, and where “target = non-nix". See for example this blog post on fully statically linked Haskell binaries or tweag/skyscope#94. See also #291 which aims to better document deployment models such as the one from Daml that you link.

This is where I think the distinction comes into play: I find these examples of support to be of the kind where rules_nixpkgs does not impede the implementation, but does not provide it either, for a couple of reasons:

1. A fully statically linked binary built with nix toolchains does not depend on a nix system, but that's a characteristic of the output, not of the toolchain / system configuration. It's more similar to the use case of a target agnostic file as the output, for example a container image or any other non-executable target agnostic data file.
2. It requires the user to implement the logic to produce the fully statically linked binary on the nix side. You cannot point rules_nixpkgs to an arbitrary nix derivation and expect the binary output to be built for a non-nix target, even if your target platform doesn't have the nix constraint.
3. Improving the documentation alone does not provide the capability to the user, it's helpful, but the user is still expected to implement that capability in their code. It's not the same as having a rule implemented in the rule set to provide that capability.

About the case where “exec = non-nix”. I think this reflects a very specific use-case that requires significant divergence from standard nixpkgs. Most Nix derivations will have run-time dependencies into the Nix store. This includes compiler toolchains. Runtime dependencies of compilers translate to “exec” platform constraints in Bazel. So, to support this use-case (as I understand it) you will require a lot of custom overrides and patches to avoid such Nix store path runtime dependencies.

Absolutely agree.

To be clear, I’m not saying you shouldn’t do this. But, I don’t think we can force all rules_nixpkgs users to carry that burden.

I agree that we should not expect users of rules_nixpkgs to carry that burden. But this is the crux of my request for clarification: are the users not expected to carry that burden because:

1. rules_nixpkgs is never expected to be used that way, and might get in the way or not, depending on implementation details
2. rules_nixpkgs is sometimes expected to be used that way, it does not provide any capability for it, but it will not get in the way
3. rules_nixpkgs is mostly expected to be used that way, it provides all the capabilities required for the use case

Clarifying which of the three helps the users who are looking for that capability decide whether they should:

1. not use rules_nixpkgs at all
2. use rules_nixpkgs as a building block for their implementation of the capability
3. report bugs / submit PRs for the capability in rules_nixpkgs

[uri-canva]

Note that host, exec, target is just how I think of the capability support matrix. The important thing is having such a support matrix defined, I'd be perfectly happy with having it defined in other terms.

[uri-canva]

To give an example of a capability rules_nixpkgs provides, https://github.com/tweag/rules_nixpkgs/blob/420370f64f03ed9c1ff9b5e2994d06c0439cb1f2/toolchains/cc/cc.nix allows you to use cc from nixpkgs in bazel. You're not expected to implement this wrapping logic in your own nix code, and you're not expected to write any special starlark / custom bazel rule on the consuming side.
To me that's very different from some of the other examples of things you can do with rules_nixpkgs.

[uri-canva]

rules_nixpkgs on the other hand is about providing fine-grained access to Nix provided toolchains and other dependencies to a Bazel project. [...] So, I'd view those as separate use-cases and would not view buildBazelPackage as a replacement for rules_nixpkgs.

Makes sense.

To clarify, rules_nixpkgs ”support” of this use-case only extends so far as to get out of the way in this case. E.g. to support a Unix + Windows cross-platform project.

Makes sense, this is an example of supporting only in so far as not being an impediment.

[uri-canva]

On i., yes, we’re working on it. On ii., can you clarify in which way this fails for you? On iii. could you also clarify? Usually NixOS is the problem here, not rules_nixpkgs.
In which way does this fail for you? I would expect an incompatibility between the Nix provided C toolchain and the downloaded Go toolchain. But, that falls outside of rules_nixpkgs.

I'll create a separate issue to go into the details once we've clarified what kind of support rules_nixpkgs is expected to have for this use case, so that we both know what the desired outcome of the discussion on the specific issue is: PR to fix things in rules_nixpkgs / changes on my side on how I use it / mix of both or neither.

[uri-canva]

Usually NixOS is the problem here, not rules_nixpkgs.
I would expect an incompatibility between the Nix provided C toolchain and the downloaded Go toolchain. But, that falls outside of rules_nixpkgs.

This might play a big role in the support matrix, though I don't know how we could explain it / document it. From the perspective of a bazel user, one might expect rules_nixpkgs to abstract these nix idiosyncrasies away from them, so it's important to clarify rules_nixpkgs does not handle this.

This is a problem in a lot of tooling that builds on top of nix: many of them highlight the advantages of using nix based tooling in your development workflow, very few explain the implications that has for which platforms you can target, and what other parts of the non-nix development ecosystem you can integrate with.

[uri-canva]

I have more points about what I would like rules_nixpkgs to support / aim for / do from a user's perspective, but I'll refrain from that until we've clarified this further, so I don't explain something that is completely detached from the design of rules_nixpkgs.

[uri-canva]

One last thing I want to highlight: the support matrix of rules_nixpkgs as it is now and the ideal support matrix rules_nixpkgs is striving towards could be very different. If they are I think it's important to have both, since a user would care mostly about the first one, but a contributors will care mostly about the latter one.