Poetry doesn't handle developmental releases

[hoefling]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 32

• Poetry version: 1.0.10 / 1.1.b3 / master are all affected

• Link of a Gist with the contents of your pyproject.toml file:

[tool.poetry]
name = "fizz"
version = "0.1.0"
description = ""
authors = ["oleg.hoefling <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"
tf-estimator-nightly = "^2.4.0-alpha.2020092501"
tf-nightly = "^2.4.0-alpha.20200924"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

To reproduce from scratch:

$ poetry new fizz
$ cd fizz/
$ $ poetry add tf-nightly
Creating virtualenv fizz-y_Gqf9Xy-py3.8 in /home/oleg.hoefling/.cache/pypoetry/virtualenvs
Using version ^2.4.0-alpha.20200925 for tf-nightly

Updating dependencies
Resolving dependencies... (0.8s)

[SolverProblemError]
Because no versions of tf-nightly match >2.4.0-alpha.20200925,<3.0.0
 and tf-nightly (2.4.0.dev20200925) depends on tf-estimator-nightly (*), tf-nightly (>=2.4.0-alpha.20200925,<3.0.0) requires tf-estimator-nightly (*).
So, because no versions of tf-estimator-nightly match *
 and fizz depends on tf-nightly (^2.4.0-alpha.20200925), version solving failed.

This is the first problem, looks like the same issue though: tf-estimator-nighlty dependency can't be resolved. Adding it manually seems to be a quick workaround:

$ poetry add tf-estimator-nightly
Using version ^2.4.0-alpha.2020092501 for tf-estimator-nightly

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file


Package operations: 10 installs, 0 updates, 0 removals

  ...
  - Installing tf-estimator-nightly (2.4.0.dev2020092501)

Note the discrepancy between versions: the correct one is installed (2.4.0.dev2020092501), the one recorded in pyproject.toml (2.4.0-alpha.2020092501) doesn't exist.

Now adding the actual dependency works:

$ poetry add tf-nightly
Using version ^2.4.0-alpha.20200925 for tf-nightly

Updating dependencies
Resolving dependencies... (1.2s)

Writing lock file


Package operations: 33 installs, 0 updates, 0 removals

  ...
  - Installing tf-nightly (2.4.0.dev20200925)

Inspecting the metadata of a wheel built with poetry build:

...
Requires-Dist: tf-estimator-nightly (>=2.4.0-alpha.2020092501,<3.0.0)
Requires-Dist: tf-nightly (>=2.4.0-alpha.20200925,<3.0.0)

An attempt to install the wheel now fails:

$ python3 -m venv /tmp/tst
$ /tmp/tst/bin/pip install dist/fizz-0.1.0-py3-none-any.whl
Processing ./dist/fizz-0.1.0-py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement tf-nightly<3.0.0,>=2.4.0-alpha.20200925 (from fizz==0.1.0) (from versions: 2.2.0.dev20200501, 2.2.0.dev20200502, 2.2.0.dev20200503, 2.2.0.dev20200504, 2.2.0.dev20200505, 2.2.0.dev20200506, 2.2.0.dev20200507, 2.2.0.dev20200508, 2.3.0.dev20200512, 2.3.0.dev20200513, 2.3.0.dev20200514, 2.3.0.dev20200515, 2.3.0.dev20200516, 2.3.0.dev20200518, 2.3.0.dev20200519, 2.3.0.dev20200521, 2.3.0.dev20200522, 2.3.0.dev20200523, 2.3.0.dev20200524, 2.3.0.dev20200525, 2.3.0.dev20200526, 2.3.0.dev20200527, 2.3.0.dev20200528, 2.3.0.dev20200529, 2.3.0.dev20200530, 2.3.0.dev20200531, 2.3.0.dev20200601, 2.3.0.dev20200602, 2.3.0.dev20200603, 2.3.0.dev20200604, 2.3.0.dev20200605, 2.3.0.dev20200608, 2.3.0.dev20200609, 2.3.0.dev20200610, 2.3.0.dev20200611, 2.3.0.dev20200612, 2.3.0.dev20200613, 2.3.0.dev20200614, 2.3.0.dev20200615, 2.3.0.dev20200616, 2.3.0.dev20200617, 2.3.0.dev20200618, 2.3.0.dev20200619, 2.3.0.dev20200620, 2.3.0.dev20200621, 2.3.0.dev20200622, 2.3.0.dev20200623, 2.3.0.dev20200624, 2.3.0.dev20200625, 2.4.0.dev20200630, 2.4.0.dev20200701, 2.4.0.dev20200702, 2.4.0.dev20200703, 2.4.0.dev20200704, 2.4.0.dev20200705, 2.4.0.dev20200706, 2.4.0.dev20200707, 2.4.0.dev20200708, 2.4.0.dev20200709, 2.4.0.dev20200710, 2.4.0.dev20200711, 2.4.0.dev20200712, 2.4.0.dev20200715, 2.4.0.dev20200716, 2.4.0.dev20200717, 2.4.0.dev20200718, 2.4.0.dev20200719, 2.4.0.dev20200720, 2.4.0.dev20200721, 2.4.0.dev20200722, 2.4.0.dev20200723, 2.4.0.dev20200724, 2.4.0.dev20200725, 2.4.0.dev20200726, 2.4.0.dev20200727, 2.4.0.dev20200728, 2.4.0.dev20200729, 2.4.0.dev20200730, 2.4.0.dev20200731, 2.4.0.dev20200801, 2.4.0.dev20200802, 2.4.0.dev20200803, 2.4.0.dev20200805, 2.4.0.dev20200806, 2.4.0.dev20200807, 2.4.0.dev20200808, 2.4.0.dev20200809, 2.4.0.dev20200810, 2.4.0.dev20200811, 2.4.0.dev20200812, 2.4.0.dev20200813, 2.4.0.dev20200815, 2.4.0.dev20200816, 2.4.0.dev20200817, 2.4.0.dev20200818, 2.4.0.dev20200819, 2.4.0.dev20200820, 2.4.0.dev20200821, 2.4.0.dev20200822, 2.4.0.dev20200823, 2.4.0.dev20200824, 2.4.0.dev20200825, 2.4.0.dev20200826, 2.4.0.dev20200827, 2.4.0.dev20200828, 2.4.0.dev20200829, 2.4.0.dev20200830, 2.4.0.dev20200903, 2.4.0.dev20200904, 2.4.0.dev20200905, 2.4.0.dev20200906, 2.4.0.dev20200907, 2.4.0.dev20200908, 2.4.0.dev20200909, 2.4.0.dev20200910, 2.4.0.dev20200911, 2.4.0.dev20200912, 2.4.0.dev20200913, 2.4.0.dev20200914, 2.4.0.dev20200915, 2.4.0.dev20200916, 2.4.0.dev20200917, 2.4.0.dev20200918, 2.4.0.dev20200919, 2.4.0.dev20200920, 2.4.0.dev20200921, 2.4.0.dev20200922, 2.4.0.dev20200923, 2.4.0.dev20200924, 2.4.0.dev20200925)
ERROR: No matching distribution found for tf-nightly<3.0.0,>=2.4.0-alpha.20200925 (from fizz==0.1.0)

The reason for this is that 2.4.0-alpha.20200925 is normalized to 2.4.0a20200925 according to PEP 440, which is a prerelease, not a developmental release. Surely, "publishing developmental releases on a public index is discouraged", but still possible, and this is one example of resulting dependency resolution failure.

[finswimmer]

Hello @hoefling,

can you try it with poetry 1.1.0rc1 please?

fin swimmer

[hoefling]

Hi @finswimmer, indeed, adding dev releases to dependencies works with 1.1.0rc1, e.g. poetry add tf-nightly raises no errors! However, the wrong metadata in built wheels is reproducible with 1.1.0rc1, so the wheels with a tf-nighlty dependency built with 1.1.0rc1 are still not installable. The current workaround I'm using right now is modifying the resulting wheel -- alpha is replaced with dev in METADATA and METADATA's hash is rewritten in RECORD.

[joshorr]

@finswimmer As a FYI, in case it's helpful: I am having this same problem with the latest master (as of right now):

Because ... depends on djangorestframework-filters (1.0.0.dev2) which doesn't match any versions, version solving failed.

Have this in my pyproject.toml file:

djangorestframework-filters = "1.0.0.dev2"

This works on poetry 1.1.6, but not on master.

[abn]

@joshorr the combination of the above two PRs should fix the issue.

$ poetry add [email protected]

Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing asgiref (3.3.4)
  • Installing pytz (2021.1)
  • Installing sqlparse (0.4.1)
  • Installing django (3.2)
  • Installing django-filter (2.4.0)
  • Installing djangorestframework (3.12.4)
  • Installing djangorestframework-filters (1.0.0.dev0)

[abn]

These PRs should also resolve this issue.

$ poetry add tf-nightly
Creating virtualenv fizz in /tmp/fizz/.venv
Using version ^2.6.0-dev.20210422 for tf-nightly

Updating dependencies
Resolving dependencies... (26.0s)

Writing lock file

Package operations: 38 installs, 0 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0)
  • Installing idna (2.10)
  • Installing pyasn1 (0.4.8)
  • Installing urllib3 (1.26.4)
  • Installing cachetools (4.2.1)
  • Installing oauthlib (3.1.0)
  • Installing pyasn1-modules (0.2.8)
  • Installing requests (2.25.1)
  • Installing rsa (4.7.2)
  • Installing setuptools (56.0.0)
  • Installing six (1.15.0)
  • Installing google-auth (1.29.0)
  • Installing requests-oauthlib (1.3.0)
  • Installing absl-py (0.12.0)
  • Installing google-auth-oauthlib (0.4.4)
  • Installing grpcio (1.37.0)
  • Installing markdown (3.3.4)
  • Installing numpy (1.19.5)
  • Installing protobuf (3.15.8)
  • Installing tensorboard-data-server (0.6.0)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing werkzeug (1.0.1)
  • Installing wheel (0.36.2)
  • Installing astunparse (1.6.3)
  • Installing flatbuffers (1.12)
  • Installing gast (0.4.0)
  • Installing google-pasta (0.2.0)
  • Installing h5py (3.1.0)
  • Installing keras-nightly (2.6.0.dev2021042200)
  • Installing keras-preprocessing (1.1.2)
  • Installing opt-einsum (3.3.0)
  • Installing tb-nightly (2.5.0a20210419)
  • Installing termcolor (1.1.0)
  • Installing tf-estimator-nightly (2.6.0.dev2021042201)
  • Installing typing-extensions (3.7.4.3)
  • Installing wrapt (1.12.1)
  • Installing tf-nightly (2.6.0.dev20210422)

[joshorr]

@abn that worked! Thank you so much!!!! When using gemfury (3rd party private pypi) with this latest master, it's much faster (80 seconds vs 19 seconds to resolve everything via poetry update) then using 1.1.6. That's why I am interested in using this.

The other other thing that's not working is when a version requirement is like this: python-dotenv = "^0". It errors out with this message:

Because ... depends on python-dotenv (^0) which doesn't match any versions, version solving failed.

This works on 1.1.6, see this poetry doc about "^0":

https://python-poetry.org/docs/versions/

I have a work around though, I just do a ">=0.0.0,<1.0.0" instead, that works.

I'll try to make a new issue if you desire later today. I don't see any other issues about the "^0" problem.

[abn]

Maybe raise another issue with the details so we can figure out if it's an issue. Happy to take a look :)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.