Allow to get rid of nix develop "shell" logic

[roberth]

Is your feature request related to a problem? Please describe.

Remove the tight coupling with stdenv by defining a small interface

We have a number of bugs and feature requests related to nix-shell and nix develop.

"Shells" appear to be built on derivations, but they are not a derivation feature. A derivation has a builder, which is a self-contained, non-interactive process that's written for a single purpose.

Currently, to create a shell, Nix has to intrude on stdenv's domain and make assumptions about how the builder is implemented in order to modify the derivation to support some specific code it needs to run.

This creates a situation where nix develop and stdenv are tightly coupled, which holds back both components' development and does not allow alternative derivation builders to support shells.

To break this tight coupling, I propose a new interface between nix develop and the expression language. Instead of taking apart a derivation, we can ask the expression nicely to produce a command that launches a shell for us.

Concretely, this means nix develop will perform the following pseudocode

nix develop () {
  if pkg?devShell {
    nix run pkg.devShell
  } else {
    nixDevelopLegacy(pkg)
  }
}

EDIT: additionally, see below for isDevShell

By delegating the construction of a shell process to stdenv, we empower Nixpkgs to solve its own problems. This has the additional benefit that Nixpkgs allows easier collaboration, breaking changes and more risk taking in general. After all nix has a much stronger expectation of compatibility, because nix can not be pinned to projects.

Other benefits include

• shells become first class in Nixpkgs, which is the underlying goal of provide getDerivationEnvironment as a primop to reference nix develop drv from Nix code #7468 ("XY problem")
• implementing a minimal shell is trivial mkShellMinimal: Create an ultra minimal nix-shell nixpkgs#132617
• switching to a different builder shell becomes a possibility https://github.com/nixpkgs-architecture/issues/issues/7
• support another dev shell Support non-bash shells in nix-shell. #545

Out of scope for this change (but relevant)

• Producing an accurate sandbox-like environment for debugging builds. This has traditionally been a task of nix-shell, but it just doesn't produce a sandbox environment, leading to questions, typically: "why does this work in nix-shell, but not in my build?". This should be solved by a different command that is capable of producing an actual sandbox environment. This is a significantly different task that needs its own solution. It does also need to interface with user code, but in this case it's the derivation, not the expression that should be asked. This slightly special sandbox may be differentiated by only the existence of a tty on stdin, or perhaps also a single special environment variable. It will be up to the builder process to read these clues and kick into a debugging mode; typically an interactive shell.

Describe the solution you'd like

See pseudocode above.

Similarly:

nix print-dev-env() {
  if drv.isDevShell or false {
    nix run drv -- --print-dev-env
  } else if drv?devShell {
    nix run drv.devShell -- --print-dev-env
  } else {
    nixPrintDevEnvLegacy(drv)
  }
}

Describe alternatives you've considered

Keep status quo, expend Nix team effort on shell features that the community can more easily fix themselves, and leave more important Nix issues and improvements behind.

Additional context

Solution to

• provide getDerivationEnvironment as a primop to reference nix develop drv from Nix code #7468
• most "nix-shell" issues: https://github.com/NixOS/nix/issues?q=is%3Aopen+is%3Aissue+label%3Anix-shell

Possible nixpkgs-side devShell implementation stub

• POC: devShell interface nixpkgs#206728

Pushes nix develop toward the dev side of

• Split nix-shell logic #777

Priorities

Add 👍 to issues you find important.

[fricklerhandwerk]

Discussed in Nix team meeting 2022-12-23:

• agreement that this is worth discussing more
• @edolstra a nix run-like interface is not sufficient; would like to make the expression export a bash script and/or plain environment variables
• @edolstra: multi-shell support is actually possible by providing aliases that call into bash
  • @roberth: this is a never ending pit of feature requests, and a feature/tool that's beneficial to users outside the Nix ecosystem, so it should be its own project that Nixpkgs can use to implement this other shell feature

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-23-nix-team-meeting-minutes-19/24400/1

[tomberek]

develop does a few things in order:

1. analyze the derivation and create a build environment for the derivation
2. initialize the build environment using the "--rcfile" trick in order to go last, juggle some env vars for convenience, and call hooks

Most of the non-nix debug usages care about (2), and we've built tooling to sidestep or undo the effects of (1), eg: mkShell, devshell, and so forth. My interpretation of this proposal is to make it easier and less of a hack to bypass (1).

[tomberek]

Steps ahead

• idea is approved
• need nix develop and nix print-dev-env interface/specification descriptions
• implementation, POC -> MVP -> experimental -> complete

Random notes:

• nix debug

• breakpointHook - cntr

• nix bundle --bundle builderInSandbox

• nix bundle --bundle builderInOCI

• nix bundle --bundle in QEMU

• nix develop --with nixpkgs#make-me-a-devShell

• nix develop --with custom#pkgs.defaultDevShell

• allow for overriding of get-env.sh

  • passthru.devShell
  • passthru.devShell { drv = builtins.getDerivationEnvironment drv; get-env = "source $something"; }
  • meta.get-env?
  • passthru.devShell = self // { builder = "source $my-own-stdenv";};
  • passthru.devShell = mkShell { name = "asdfas"; builder = "something-custom";}
  • passthru.devShell = drv: function-that-creates-JSON

[fricklerhandwerk]

Discussed in the Nix team meeting 2023-01-23:

• nix develop is breaking abstraction wrt Nixpkgs. Try to decouple that
• @edolstra: We shouldn't require the user to have to define what it shell env should look like, it should be transparent
• The “user” in here would be Nixpkgs's stdenv which is already the one providing the environment, so it seems reasonable for it to be defined here
• A few other implementations of that logic exist
  • Nix-shell like Docker conatiners
  • .inputDerivation
• (more discussion)
• Decision:
  • Add some documentation for the proposed protocol
  • Finish the implementation

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-01-23-nix-team-meeting-minutes-26/25433/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-04-14-nix-team-meeting-minutes-48/27358/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-03-nix-team-meeting-minutes-68/30337/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakey-profile-declarative-nix-profiles-as-flakes/35163/3

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-doesnt-nix-collect-garbage-collect-all-the-garbage/38587/16

[fricklerhandwerk]

Related: #4702 (comment)

[roberth]

@fricklerhandwerk, in #4702 (comment) you propose to

reduce the Nix command line interface to essentially nix run, nix build, nix eval, and nix store

Did you mean that list to be exhaustive - ie discourage even a simple nix develop entrypoint, or did you omit it not to balloon the scope there?

I believe this extra entrypoint - nix develop - is extremely valuable as a "thin waist" around which Nix-aware tooling can interoperate with Nix-aware projects.

[fricklerhandwerk]

Not exactly exhaustive, but a good approximation. I haven't made up my mind on nix develop: It is centered around stdenv and the idea that the build environment is somehow run by a shell. That's was done historically for good practical reasons, but it's not a hard requirement. Migrating away from that surely has a cost. And of course Nix could have nix develop start any (even the current) shell with the builder's environment as defined by the store derivation, but how useful would that really be? In addition it's hard to even imagine how to pull off such a migration purely on the Nix side, as Nixpkgs would have to follow suit... what would that entail for backward and forward compatibility? A lot of stdenv happens within the build script. Arguably the whole notion of a shell environment for derivation builds is native to stdenv.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-cli-working-group-member-search/30517/22

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-04-10-nix-team-meeting-138/43585/1