Use Poetry & Poetry2nix for environment and plugin management

[adisbladis]

This is a proposal to adopt the Python package manager Poetry as a dependency manager and poetry2nix as the Nix integration.

Introduction to Poetry

Poetry is a tool to handle dependency management & building of Python packages.
It has a well written dependency solver, an active community & a nice consistent CLI.

Poetry aims to improve on Python packaging by using PEP 517 (better known as pyproject.toml) instead of setup.py and can aggregate a number of separate configuration files for python related tooling into this one declarative file.

Poetry is quickly getting traction in the wider Python community, an example of a high-profile open source project that has started to adopt Poetry is the code formatter Black.

Introduction to poetry2nix

Poetry2nix is a lock-file based converter that creates derivations on the fly without the need to write expressions for every single package.

This is done completely pure and works in restricted eval with the caveat that git dependencies uses builtins.fetchGit by default.
To combat this issue poetry2nix also provides a CLI to generate an overlay function with pinned git dependencies using pkgs.fetchgit and an explicit sha256.
This code generation is minimal and will only include packages that would have otherwise been fetched with builtins.fetchGit.

While poetry2nix hasn't seen much use in the open multiple companies are at various stages of adoption in anywhere from testing phases to production.
Poetry2nix is included in nixpkgs, and the Poetry package manager itself is packaged using poetry2nix.

Disclaimer: I'm the author of this tool.

Why NixOps needs a dependency solver

With the new plugin infrastructure we have a brand new dependency problem.
Plugins depends on NixOps and may or may not be compatible with a given version of NixOps & the other way around.

This is fantastic news but complicates packaging and in particular the story around how we get this into nixpkgs.
For this Plugins need to be able to express minimum and maximum compatible Nixops versions.

Plugins themselves also have dependencies that we need to get in the final nixpkgs NixOps package, the status quo is that all of these dependencies need to be in nixpkgs.
The constraint of having packages be a part of the nixpkgs python set potentially makes backporting NixOps and plugin releases hard.

Developer experience

Let's start this segment off by clearly stating that unless you want to modify the dependencies of the project you do not have to use Poetry directly.
To start hacking simply enter a nix-shell and start hacking.

If you want to you can use Poetry's virtualenv integration, though my personal favourite workflow is to use Poetry for what it's good at, namely dependency management & use Nix for what it's good at, namely package management.

For plugin authors all of these changes means that they can hack on their NixOps plugin just as if it was any other completely normal Python package.

Development scenarios

• Scenario 1

plugin$ nix-shell
- mutable plugin source, immutable nixops src
- mypy works
- tests work
- `nixops` works, using this plugin

• Scenario 2

nixops$ nix-shell
- no plugins
- mutable nixops src
- mypy works
- tests work

• Scenario 3

plugin$ nix-shell
- mutable plugin source, mutable nixops src
- mypy works
- tests work
- `nixops` works, using this plugin

Works with a custom override pointing the plugin development shell at the nixops source, or by changing pyproject.toml to point at a local path and running poetry install.

Major changes

This proposal removes all plugins from the main NixOps repository as these are devoloped & tested independently of NixOps.
It's possible we may want to have a separate set of integration tests doing actual deployments with multiple plugins.

Showcases

I have pushed this (the main PR), a branch where the nixops-aws plugin is converted and a PoC how this will all work together:

• Nixops-aws
• nixpkgs - Note that this is only a rough outline of how I think we should manage nixops + plugins.

FAQ (or at least questions I expect will come)

• Does my plugin have to use Poetry?

No. While I would encourage more usage of Poetry it's perfectly fine to use setuptools/distutils/flit or whatever other solution you'd like.

• What about Pipenv? Isn't that recommended?

Pipenv is the recommended packaging solution in the official PyPa docs though this (from what I've been able to observe) was a mostly political move by the Pipenv author and a very premature recommendation.
It is full of subtle and not so subtle bugs, it's dependency solver is fundamentally flawed.
Pipenv has currently not had a single commit on it's master branch for 4 months and no release since November 2018. At this point the project can only be assumed to be dead, and even if it wasn't it's not mature enough to adopt.

There is an open issue by one of the Python core devs to remove this recommendation.

• What about using Flakes?

Flakes is in my opinion not a suitable contender. For one flakes do not have a dependency solver and it means we need to rely on pythonPackages from nixpkgs.
This will make developing plugins a lot harder, a dependency might not even be packaged yet, a version of a dependency may be too old or too new.
That said Nixops + a solved plugin environment could be provided as a flake.

Next steps

If this looks good I will go ahead and start cleaning up things we no longer need from the repositories and come up with a finalised version of this WIP.
I have purposely tried not to change too much in this PR so it will be easier to get a grasp of what's actually new.

cc @AmineChikhaoui @grahamc @disassembler @johnalotoski

[grahamc]

I've rebased this to fix a merge conflict in release.nix.

[domenkozar]

Nice!

Why is NixOS test removed?

Manual should still be built and included.

[adisbladis]

@domenkozar According to 21dde70 the test is broken for a long time.
I'd very much like for us to have more NixOS tests in the future, but for now we opted to remove it.

[grahamc]

Should we also move the nix/*.nix files in to the nixops directory, to match plugins? That seemed to be cleaner, anyway.

[grahamc]

You know, don't worry about it on this PR. I don't want this PR to be a catch-all for little improvements. That can be next PR :)

[grahamc]

fwiw I'm not going to be a stickler on the ratchet for this PR. If you improve it, great. If not, ok too.

[grahamc]

To merge this, probably should do:

• Poetry plugin docs adisbladis/nixops#4
• update nixops-aws (NOT a py3 port; just a plugin update)
• update nixops-hetzner (NOT a py3 port; just a plugin update) -- Introduce poetry(2nix), GitHub Actions, blackify all the files nixops-hetzner#9 Note: needs to un-pin the poetry2nix dep before ticking this box.
• update nixops-vbox py3 updated, plugin update: Poetry2nixify nix-community/nixops-vbox#3
• update nixops-libvirtd py3 updated, plugin update: Poetry2nixify nix-community/nixops-libvirtd#10
• update nixops-datadog py3 updated, plugin update: https://github.com/nix-community/nixops-datadog/pull/1/files
  to verify the instructions work. Ideally, a handful of people would do the plugin update work to have a few different people run through them.

An open question: @adisbladis, you have work on a nixopsWithPlugins. Maybe this should be added to nixpkgs as nixops2Unstable so people can start exercising it? I'm not sure, what do you think?

[adisbladis]

@grahamc nixops-aws is "pluginized" in https://github.com/adisbladis/nixops-aws/tree/poetry2nix-plugins

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update/6525/1

[grahamc]

Having working with this PR on several plugins and deployments, it has been really useful and great, and I am excited to smash this merge button. I think there are some other improvements to make, but I really appreciate how good this PR is already. For example, I have a few patches to the authoring.rst I wrote. Thanks, @adisbladis, for building such a great tool (poetry2nix) and showing us the light on how to do python packaging better than before.