Platform autodetection

[saksmt]

Since there is currently no support for custom platform/constraint autodetection/autodiscovery (see bazelbuild/bazel#8766) there is no way sane way to use neither nixpkgs_python_configure nor nixpkgs_sh_posix_configure and they just being autodetected by bazel itself from existing environment and not from nixpkgs (at least until no --platforms and host_platform equal to @io_tweag_rules_nixpkgs//constraints:linux_x86_64_nixpkgs specified manually)

So should we just drop that constraints from toolchains?

Pros:

• fully hermetic nix-driven builds with zero fuss
• as I can understand from documentation to toolchains and constraints/platforms it wasn't meant to be used in toolchains like this anyway (but probably we whould wrap nix itself in toolchain and provide it under constraint - that would make sense, but not until bazel figures out how to automatically provide values for constraints based on some user code)

Cons:

• currently calling those _configure ultimately does nothing, so we may break something with that change (do we really care? especially if it is called directly and MEANT to do so?)
• from the doc it is not clear if those _configure are public api of this rules or not, so probably this issue is for nothing (we should write some doc probably)

[aherrmann]

there is no way sane way to use neither nixpkgs_python_configure nor nixpkgs_sh_posix_configure and they just being autodetected by bazel itself from existing environment and not from nixpkgs (at least until no --platforms and host_platform equal to @io_tweag_rules_nixpkgs//constraints:linux_x86_64_nixpkgs specified manually)

You can add the following to .bazelrc (or .bazelrc.local):

build --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/constraints:linux_x86_64_nixpkgs

or

build --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/constraints:darwin_x86_64_nixpkgs

to configure this without having to manually set these flags all the time.

So should we just drop that constraints from toolchains?

Unfortunately, that doesn't work for all required use-cases. An example is rules_haskell, where we want to test on multiple platforms with and without Nix. If we're testing with Nix we want to use the toolchain defined by nixpkgs_sh_posix_configure and if we're testing without Nix we want to use the toolchain defined by sh_posix_configure.

The Bazel docs specify

Available platforms and toolchains are tracked as ordered lists for determinism, with preference given to earlier items in the list.

I.e. the first toolchain fitting all constraints wins. Consider the following order of toolchain definitions:

nixpkgs_sh_posix_configure(...)
sh_posix_configure()

If we didn't have the nixpkgs constraint, then both those toolchains would have the same set of constraints and Bazel would always choose the nixpkgs one. However, this would fail in cases where Nix is not available. To avoid this issue we require the additional nixpkgs constraint on the nixpkgs provided toolchain.

• as I can understand from documentation to toolchains and constraints/platforms it wasn't meant to be used in toolchains like this anyway

I don't follow, could you point to the relevant part of the docs? The platforms documentation explicitly mentions the use-case of different system configurations and tool versions.

(but probably we whould wrap nix itself in toolchain and provide it under constraint - that would make sense, but not until bazel figures out how to automatically provide values for constraints based on some user code)

Sadly, this is not possible. Toolchains are only available in regular rules, i.e. for targets defined in BUILD files. However, nixpkgs_package and similar are repository_rules. At that stage toolchain resolution has not happened, yet.

• from the doc it is not clear if those _configure are public api of this rules or not, so probably this issue is for nothing (we should write some doc probably)

Thanks for pointing that out. They are, the docs need to be fixed.

[saksmt]

You can add the following to .bazelrc

This approach actually don't scale at all since we may have another toolchain that requires specific constraint which leads to another platform that can't be combined easily, also it's a bit uncomfortable to manually set platform (which includes not only nix, but whole environment). Probably it would be better to define config_setting and try to work it out that way (at least it scales well).

Unfortunately, that doesn't work for all required use-cases

Is it possible to check if environment contains nix-build before registering toolchain? That would solve this issue.

I don't follow, could you point to the relevant part of the docs?

Probably it's just me, but as I could understand the purpose of that is to define capabilities of system for toolchain to work correctly, i.e. if python requires glibc version 2.26 to run that would be constraint - in my brain it is about runtime, not configuration/installation time. In nix this concept is missing, but if you ever encountered ebuilds: in my mind constraints are about RDEPEND and here we use it for DEPEND.

[mboes]

Probably it would be better to define config_setting and try to work it out that way (at least it scales well).

Related: bazelbuild/platforms#10.

[aherrmann]

You can add the following to .bazelrc

This approach actually don't scale at all since we may have another toolchain that requires specific constraint which leads to another platform that can't be combined easily,

Yes, if you need to combine platforms then you will have to define a custom platform yourself. Bazel platform definitions support inheritance which may reduce the required boilerplate a bit. Unfortunately, the docs point out the following

Platforms may use the parents attribute to specify another platform that they will inherit constraint values from. Although the parents attribute takes a list, no more than a single value is currently supported, and specifying multiple parents is an error.

So, this doesn't allow you to compose platforms, only to adorn a platform with additional constraints. However, for rules_nixpkgs that should be enough. E.g.

platform(
    name = "xyz_with_nixpkgs",
    parents = [":xyz"],
    constraint_values = [
        "@io_tweag_rules_nixpkgs//nixpkgs/constraints:nixpkgs",
    ],
)

I don't know if there are any plans to lift that single value limitation on the parents attribute.

Probably it would be better to define config_setting and try to work it out that way (at least it scales well).

Could you clarify what you have in mind here? Config settings are consumed by select, but I don't see where to place a select in this context. Note, select cannot be used on repository rules.

Unfortunately, that doesn't work for all required use-cases

Is it possible to check if environment contains nix-build before registering toolchain? That would solve this issue.

It is possible to check if nix-build is in PATH, we currently use it for the fail_not_supported attribute. However, as explained above, without the additional nixpkgs constraint, this would mean that having nix-build in path would always cause Bazel to use a nixpkgs provided toolchain, even if someone wants to use a non-nixpkgs toolchain.

I don't follow, could you point to the relevant part of the docs?

Probably it's just me, but as I could understand the purpose of that is to define capabilities of system for toolchain to work correctly, i.e. if python requires glibc version 2.26 to run that would be constraint - in my brain it is about runtime, not configuration/installation time. In nix this concept is missing, but if you ever encountered ebuilds: in my mind constraints are about RDEPEND and here we use it for DEPEND.

As the documentation points out

Bazel recognizes three roles that a platform may serve:

Host - the platform on which Bazel itself runs.
Execution - a platform on which build tools execute build actions to produce intermediate and final outputs.
Target - a platform on which a final output resides and executes.

If I understand the ebuilds analogy correctly, then "Execution" would correspond to DEPEND and "Target" to RDEPEND. The toolchain rule has the attributes exec_compatible_with and target_compatible_with to define the corresponding constraints. So, Bazel's constraints are used for both. I think you're raising a good point, though. The use of these constraints in rules_nixpkgs is probably not as carefully thought out as it should be. But, I don't think the answer is to drop the constraint.

Sticking to the Python example: A py_binary built with a nixpkgs provided Python toolchain, will require the Python interpreter in the Nix store path at runtime, so the nixpkgs constraint should go into target_compatible_with. Currently, it is only set on exec_compatible_with.

[aherrmann]

Closing for inactivity and as there doesn't seem to be an actionable outcome.