Opaque dependency

[roberth]

Is your feature request related to a problem? Please describe.

• Nix currently can't express a runtime-only dependency.
• Runtime-only dependencies do not need to be realised for the build, and may be realised independently or concurrently, etc.
• By not realising such dependencies, CI may avoid putting some unnecessary load on the cache, despite having a real dependency. (Assuming the dependency is not tested)

Describe the solution you'd like

• Some string context modifier that turns a normal string context into one that's "opaque"
• New derivation attribute that lists which dependencies are opaque
• Two levels of sandbox functionality
  • In environments where a path can not effectively be hidden, the path won't be realised, but may be present
  • In a full featured sandbox, the opaque dependency is not available during the build
• When a path is added to a derivation both as opaque and as a regular dependency, the opaque dependency is ignored, to allow composition to just work
  • Maybe an extra flag could make opaqueness required and turns this into an error, to trade between composition and a guarantee. Perhaps this should impose the requirement on the sandbox as well

Describe alternatives you've considered

Opaque

Additional context

For exploiting the concurrency, we'll want to have two levels of validity in the store: individually valid, and closure valid. This benefits other processes as well, chiefly substitution and nix copy.

Duplicate of #1080, but perhaps a different perspective, and this issue could perhaps be more implementation oriented.

Priorities

Add 👍 to issues you find important.

[edolstra]

There is no getting around the fact that this would cause closures to behave differently depending on whether the opaque dependency is present in the Nix store, which violates a core goal of Nix.

Furthermore, Nix has no concept of "runtime-only dependencies" because there really is no such thing: e.g. the "runtime-only" dependencies of a compiler become build-time dependencies when that compiler is used in a build process.

For reducing CI load, it's probably better to have a FUSE filesystem that fetches store paths on demand.

[roberth]

Opaqueness would be a property of the relation between a derivation and another store path, not an intrinsic property of the store path.
So I don't think closures (which?) would behave all that differently - only the set of available paths in the sandbox during a build is affected. Output closures work the same.

That said I did not describe that the reference closure of the opaque dependency would also be unavailable (unless also referenced non-opaquely) - a straightforward extrapolation, it seems.

A FUSE filesystem will not be able to exploit the concurrency allowed by an opaque dependency.

If FUSE can be a requirement, I think we can also afford a sandbox that hides opaque paths (and unrelated paths) from the host store.

[thufschmitt]

Opaqueness would be a property of the relation between a derivation and another store path, not an intrinsic property of the store path.
So I don't think closures (which?) would behave all that differently - only the set of available paths in the sandbox during a build is affected. Output closures work the same.

I guess Eelco's point is based on the fact that for this to be usable, you need to be able to build a path without pulling in its runtime dependencies, and that means registering it with an incomplete closure. That might not hold true any more in the future (once/if we can separate building a path from registering it in a store), but it is indeed an issue today.

[roberth]

Yeah, so this

we'll want to have two levels of validity in the store: individually valid, and closure valid.