Imperative packagement replacement in nixFlakes aka nix profile

[Mic92]

quoted from @domenkozar in #3573 (comment)

Nice!

New subcommand nix profile for imperative package management (intended to replace nix-env). (This is not completely finished. Note that once a profile has been modified using nix profile, you can't use nix-env on it anymore.)

I really really wish we'd abandon imperative package management. I know many would disagree that it fills a gap of ease of use, but I think that can be filled with:

* `nix dev-shell` providing ad-hoc environments

* `nix flake --init pkg1, pkg2` or some command to start a flake scaffold

And in general conventions to use a flake to manage a profile and tooling around to make it really easy to start with one. Then it's just a matter of something like issuing a command to edit a profile flake.

One of the major pain points of previous interfaces was that people start using and get used to imperative package management and then there's no way to convert back and forth.

I hope it's not too late to reconsider.

[Mic92]

Maybe someone can remove the bug label. This was not my intention.

[Mic92]

Here I my thoughts about this. Imperative package management is a powerful tool for experienced user however I would encourage beginners to first see if declarative package management work for them for the following reason:

1. Impurities: Even we try hard there are still some impurities like /run/opengl-driver or breakages after glibc upgrades with $LOCALE_ARCHIVE which makes mixing different versions still brittle. Also we leave some impurities for practical reasons still open and applications in your profile might interact with other programs in your $PATH. Using declarative package management is the safest and best tested way for most users.
2. Path conflicts: Imperative package management can produce path conflicts in the profile which can be cumbersome to solve if a user different packages don't know what these files do. Declarative package management can also have these conflicts, however for many of them we specify priorities already.
3. Support: Having stateless declarative package management makes it easy for other users to reproduce the same state. This is in particular important when reporting issues. Using nix-info -m and snippets of the used configuration makes it easy for maintainers to reproduce bugs.

[domenkozar]

I would mostly like to collect what declarative package management can't do, it seems like the concerns raised can all be implemented (arguably, it's hard to say when, but if we are spending energy for next 10 years, let's do it right).

[zimbatm]

Eelco is making a valid point later in that thread: how do you install a package without being forced to make sure all the other dependencies are working as well? The larger the closure is, the more likely it is to happen.

One alternative I can think of is to be able to stack multiple profiles together. We already have the system profile and the user profile. We could imagine a scenario where the user then switches into other sub-profiles. Or even merge those together. That way the user has more control over their size. Potentially each profile could have a link-back to their configuration so that nix profile edit opens the top-most profile configuration into the editor.

[domenkozar]

Eelco is making a valid point later in that thread: how do you install a package without being forced to make sure all the other dependencies are working as well? The larger the closure is, the more likely it is to happen.

With nixpkgs pinned via flakes that's pretty much the default. You can even have multiple versions of nixpkgs (unstable + stable) to get you 100% of what I've seen people need.

I really think there's only something like nix profile --init --user missing that would create ~/config/nix/flake.nix which would describe your user profile and you just keep adding packages.

[edolstra]

"Imperative" package management, i.e. the ability to install/upgrade packages independently, is literally Nix's original use case. We should not get rid of that because then we lose one of the most compelling advantages over classical package managers.

Flakes don't help here. Yes, a flake can depend on multiple versions of Nixpkgs. But the flake still needs to be evaluated in its entirety to update a declaratively specified environment. So if you have packages from 20 different versions of Nixpkgs, Nix would first have to download those 20 Nixpkgs trees (consuming a lot of disk space) and then evaluate them. And those downloads could fail (e.g. if the upstream repo disappeared). By contrast nix-env and nix profile don't need to evaluate previously installed profile elements; profile elements are closures in the Nix store, not expressions that need to be evaluated.

[domenkozar]

"Imperative" package management, i.e. the ability to install/upgrade packages independently, is literally Nix's original use case. We should not get rid of that because then we lose one of the most compelling advantages over classical package managers.

That's a good point to keep these requirements in mind. I do think flakes allow us to get the same benefits with declarative interface and a few more bells and whistles, which I think is going to be one less environment difference to support and maintain.

Flakes don't help here. Yes, a flake can depend on multiple versions of Nixpkgs. But the flake still needs to be evaluated in its entirety to update a declaratively specified environment.

Don't flakes come with an evaluation cache? That would really speed it up and for simple (let's say 10 to 20 packages) that imperative package management is for, the penalty to pay would be quite small.

So if you have packages from 20 different versions of Nixpkgs, Nix would first have to download those 20 Nixpkgs trees (consuming a lot of disk space) and then evaluate them.

Sure, but do we want to encourage users to have software install from 20 different versions? I'm my 7 year experience with Nix, using stable and unstable does the job. With flake caching, using two versions of nixpkgs would still be almost instant to evaluate such a declarative profile.

We should encourage users to eventually upgrade their software and for example having last two stable releases in the profile is good incentive for the user to see that maybe adding another one means things are getting messy.

And those downloads could fail (e.g. if the upstream repo disappeared).

That's another drawback that is mitigated by a binary cache in my opinion. With cache.nixos.org and Cachix it shouldn't be a problem anymore.

By contrast nix-env and nix profile don't need to evaluate previously installed profile elements; profile elements are closures in the Nix store, not expressions that need to be evaluated.

It would be interesting to get some real numbers here, but with flakes cache of 2 versions if nixpkgs, I would presume evaluating 10-20 packages should be a matter of a few seconds.

---

I'm also trying to shoot myself in the foot here, as official installation for Cachix is nix-env -iA cachix -f https://cachix.org/api/v1/install, a redirect to nixpkgs.

But I'd love to if I can to use flakes to tell users how to install Cachix in declarative and always be up to date. I think that's what flakes can deliver, with some tweaks.

[edolstra]

Don't flakes come with an evaluation cache?

Yes but the cache only helps if it exists. To initialize/recreate it you still need to fetch those 20 different versions of Nixpkgs. Also, it's currently used only for top-level derivations, so it doesn't actually help if part of the environment derivation changes.

Sure, but do we want to encourage users to have software install from 20 different versions?

Yes. That's exactly what Nix was created for. If you're happy with running software from one consistent set of packages, you might as well run Ubuntu.

[zupo]

NixOS newbie here, only been using it for about a year. I switched over from Ubuntu because I like that I can come back to a server after several months and just look at one file (configuration.nix) to see what is installed.

I just lost three evenings with "my vim configuration is not applied for root" because apparently someone (probably me) ran nix-env -i vim as root when they wanted vim to edit something, before vim was installed system-wide, via configuration.nix. And this user-installed vim took precedence over the system-wide configuration.

I'm so scarred by this experience, that I now override nix-env with a script that echos Please don't do this, but rather edit /etc/nixos/configuration.nix to actively prevent any imperative installation.

[domenkozar]

It seems to me we're trading a few seconds of machine time here for potentially minutes if not hours of human time due to complexity of multiple installation paradigms on user machines.

Not to mention additional complexity of documentation, tooling, etc.

[edolstra]

It has nothing to do with machine time (though downloading potentially dozens of copies of nixpkgs is nothing to sneeze at) but whether it should be possible to install packages in isolation from each other. Given that that's Nix's original case, we're not going to remove that functionality.

[edolstra]

And in terms of complexity: nix-env -i package is way simpler than "figure out how to edit some nix expression without getting a syntax error and then run some command".

[domenkozar]

but whether it should be possible to install packages in isolation from each other.

That can be done with multiple nixpkgs inputs via flakes (or even plain Nix), at the expense of machine time.

And in terms of complexity: nix-env -i package is way simpler than "figure out how to edit some nix expression without getting a syntax error and then run some command".

The complexity comes in from managing state, there are N profiles all over the machine, some managed with declarative configuration, some with imperative. Then user wonders, why is my vim version so old?! That's where it gets messy. And exactly this ease of getting started gets people attracted and messes up their machines configuration transparency.

[Mic92]

A different problem about the current implementation of nix-env is that it uses the version number for upgrading (nix-env -u). Newer is not always better. Sometimes we have downgrades for a reason. The update might have broke something. Sometimes we also switch from pre-releases to stable versions or the other way around. Right now we use an -unstable-<date> notation for unstable packages. There is no way to communicate this change since nix will not switch from a stable version to an unstable one. We also cannot change it to -<date>-unstable because than nix-env will always stuck on the unstable version and we cannot change back.

[Kha]

And sometimes we might want to update a package even if its version has stayed the same, both to reduce total closure size and to fix impure dependencies such as on an updated system libGL. These issues finally convinced me to avoid nix-env completely in favor of home-manager.

[Mic92]

And sometimes we might want to update a package even if its version has stayed the same, both to reduce total closure size and to fix impure dependencies such as on an updated system libGL. These issues finally convinced me to avoid nix-env completely in favor of home-manager.

This one can actually fixed by adding --leq when upgrading packages. However this options is not the default hence unaware user might miss security updates, also see: #446

[Valodim]

There is a bit of a spectrum here, between keeping the feature or not. Perhaps imperative configuration could be explicitly enabled via declarative configuration? That way it would be clear where the boundary is between a purely configured system, and one that has imperative stuff configured in its profile.

At the moment, nix leans a bit towards nuding the user towards nix-env, e.g. command-not-found tells you to use it to install a package. I would personally welcome moving away from that a bit. Imperative isolated package management might have been the original use case of Nix, but declarative configuration is the killer feature it's known for today.

[7c6f434c]

Speaking of upgrades (and removal), I expect that a flake that, say, converts entire RubyGems.org in its own unique way might want literal package names as derivation names. Which means more name conflicts.

Maybe we should embrace the fact that right now -iA is recommended over -i, and make installation/upgrade/removal attribute-name-based by default?

[edolstra]

No, because we shouldn't make incompatible changes to nix-env at this point. nix profile is attribute name-based.

[7c6f434c]

Ah OK, I was mostly asking about not repeating the same with nix profile (although -eA/-uA would also be nice if attribute paths were saved on installation, not just names)

[davidak]

I said it before and i say it again: For the success of software (Nix) the UX is critical.

A simple task should be simple. So installing a program should be: nix install firefox.

I (as a user) don't care at all what that does in the background. I just want the program to be installed.

This imperative workflow could just write that package name to a declarative file.

This workflow probably applies mostly to Nix users on non-nixos linux and macOS? On NixOS it might be discouraged and the use of environment.systemPackages suggested. At least the onboarding (documentation) should focus on that.

nix install can just be the most basic way to use nix profile (install a package with that name from the default channel into the default profile). It would be a killer feature to be able to install unstable packages from the start (without adding an extra channel or whatever).

Make the interface so intuitive that it don't look off-putting in such a list: https://restic.readthedocs.io/en/stable/020_installation.html

[edolstra]

@davidak Currently it's nix profile install nixpkgs#restic. We could have nix install as an alias but right now we don't want to clutter the top-level namespace too much.

It would be a killer feature to be able to install unstable packages from the start (without adding an extra channel or whatever).

That's possible: nix profile install nixpkgs/master#restic

Closing this issue since we shouldn't remove non-declarative package management.

[domenkozar]

@edolstra This issue is no longer asking for imperative interface to be removed, but rather how it could be implemented.

The requirement of installing/updating packages without needing to touch other installed packages makes sense.

What I'm suggesting is to implement the imperative interface on top of declarative format.

Something like ~/nix/profile.yaml:

from: https://github.com/NixOS/nixpkgs/commit/db4cede4160f25143fd882ba92e3cc2cd5f93ae0
installed:
- git
- vim
- name: nixpkgs/master#restic
  commit: 7996fb0d4e8a438b0d621220c8cc3f327867c0d3
- tmux

So all design requirements would be satistifed while having imperative interface be built on top of declarative to ensure one can reproduce the environment on any machine.

[edolstra]

It's probably better to create a new issue, because after a lot of discussion it becomes hard to discern what's being asked.

Something like ~/nix/profile.yaml:

We almost have that, it's just called ~/.nix-profile/manifest.json instead of ~/.nix/profile.yaml. We just need a command to recreate a profile from a JSON file (nix profile import or something like that).

[domenkozar]

#4482

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-users-using-nix-not-nixos-wrong/9996/11

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-as-a-unified-format-for-profiles/29476/9