[RFC] Use Poetry for dependency management and packaging

[motohedgehog]

I've been recently looking into some aspects of Nubia development and I feel that our choice of dependency management infra was somehow rushed and I want to suggest another option that may be better for this particular project.

First of all, let me explain why I'm not happy with Pipenv.

1. We can support only one version of Python at a time with Pipenv which is unacceptable for a library. Pipenv is intended to be used by applications, not libraries, and things are unlikely to change (see Support multiple python_version pypa/pipenv#1050). Thus, the only option for supporting multiple versions with Pipenv is to remove Python version requirement and rely on runtime checks.
2. Pipenv is quite fragile when it comes to dependency resolution. Below are my most important observations.
   • Try to add Black as a development dependency and tell me about your experience 😉
     • If you made it to pre-release dependencies, then I have some bad news: you can only enable them globally, which is not what we want (see new 'pre' syntax in Pipfile pypa/pipenv#1760).
     • If you want to workaround this by using an exact version specifier, then you're still doomed as Pipenv refuses to recognise something like 18.9b0.
   • Try to check out our master branch and run pipenv sync. It could be a corrupted Pipenv.lock file or an incompatibility with a recent Pipenv version – in either case, I don't think that I want to see this kind of behaviour from a dependency management tool (it caused TravisCI builds failing #17).
3. Additional complexity imposed by the necessity of managing a dedicated configuration file along with setup.py and friends. There's a trend toward using pyproject.toml as a source of truth for Python projects configuration (see PEP518 for details), and we already leverage it for storing Black configuration.

With this in mind, I would like us to consider an alternative solution, namely Poetry.

1. Poetry uses the same (nice and flexible) versioning model for all the dependencies which makes it possible to support multiple Python versions at the same time.
2. Poetry dependency resolution is more robust (see the comparison with Pipenv here).
3. Poetry uses aforementioned pyproject.toml thus enabling us to have all the configuration in the same file (modulo lock file).
4. Poetry provides a convenient way of building and even publishing the package.

So far I've had a very pleasant experience with it, and the only minor caveat I see is that it's not available via Homebrew at the moment. I could submit a formula PR, but it's blocked on pypa/pip#5739 at the moment (although it probably will be solved in the next release).

@AhmedSoliman, @calebmarchent, I'm especially interested in your opinion on this.
If you guys find it reasonable, I'm happy to submit a PR as I have it almost ready 🙂

[calebmarchent]

Ideally, I would like us to be able to offer python-nubia regardless of dependency management tool, rather than baking it in. For example use could use a Docker container to get the pre-requisite modules ; with no need for an additional layer.

[AhmedSoliman]

I don't have personal experience with Poetry, however, your reasons sound compelling enough to experiment. I think it's a good idea to try out and let us know.

[AhmedSoliman]

As discussed. We are going with plain-old requirements.txt and setup.py for now. We can revisit this later when these tools mature and there is a solid story around supporting library use cases.