Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix futurist version conflict #4896

Closed
wants to merge 5 commits into from
Closed

Conversation

blag
Copy link
Contributor

@blag blag commented Mar 27, 2020

So, first off, here's the entire output of the pipconflictchecker command:

--------------------------------------------------
 Conflicts Detected
--------------------------------------------------
 - futurist(1.10.0) openstacksdk(>=2.1.0)

Before reading on, can you tell me what the exact problem is? No StackStorm components rely directly on either futurist or openstacksdk, so where are those dependencies coming from and why are they conflicting?

I couldn't figure it out from this output. I'll circle back around to this point in a bit.


This PR fixes the build failures caused by some transitive dependencies dropping support for Python 3.

In this case:

  • python-mistralclient depended on
    • osc-lib >= 1.8.0 (resolved to 2.0.0) which depended on
      • openstacksdk >= 0.15.0 (resolved to 0.44.0) which depended on
        • futurist >= 2.1.0 which does not support for Python 2

The latest version of futurist that supports Python 2 is version 1.10.0.

So, to fix the transitive dependency, I explicitly specify a direct dependency of opensacksdk < 0.44.0 for st2actions and st2api. This installs:

  • python-mistralclient
    • osc-lib version 2.0.0
      • openstacksdk version 0.43.0
        • futurist version 1.10.0

So that we don't break Python 2 support.

Please note that I am not a fan of this solution, it is simply the best of a few bad options. We cannot track and pin every transitive dependency like this - that's time consuming and unsustainable. This is a temporary solution and once we ship ST2 v3.2 and drop Python 2.7 support we won't need to track transitive dependencies like this as they remove support for Python 2.7.


Okay. Let's talk about the output of pipconflictchecker. It's pretty much useless unless you already know exactly what you're looking at, which I do not. Neither futurist nor openstacksdk appear in any of our requirements.txt files - they're both transitive dependencies - so it's exceedingly difficult to figure out what direct requirement/s need to be fixed, and pip-conflict-checker does absolutely nothing to give you any hints.

Instead of using pip-conflict-checker, I just used pip-compile from the pip-tools package. The output of that is illuminating. Here is the complete output of pip-compile when our dependencies have conflicts:

Could not find a version that matches futurist>=1.2.0,>=2.1.0 (from tooz==1.66.1->-r req.txt (line 64))
Tried: 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.1.2, 0.1.2, 0.2.0, 0.2.0, 0.3.0, 0.3.0, 0.4.0, 0.4.0, 0.5.0, 0.5.0, 0.6.0, 0.6.0, 0.7.0, 0.7.0, 0.8.0, 0.8.0, 0.9.0, 0.9.0, 0.10.0, 0.10.0, 0.11.0, 0.11.0, 0.12.0, 0.12.0, 0.13.0, 0.13.0, 0.14.0, 0.14.0, 0.15.0, 0.15.0, 0.16.0, 0.16.0, 0.17.0, 0.17.0, 0.18.0, 0.18.0, 0.19.0, 0.19.0, 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.21.1, 0.21.1, 0.22.0, 0.22.0, 0.23.0, 0.23.0, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.3.1, 1.3.1, 1.3.2, 1.3.2, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.9.0, 1.9.0, 1.10.0, 1.10.0
There are incompatible versions in the resolved dependencies:
  futurist>=1.2.0 (from tooz==1.66.1->-r req.txt (line 64))
  futurist>=2.1.0 (from openstacksdk==0.44.0->osc-lib==2.0.0->python-mistralclient->-r req.txt (line 19))

(The 2.1.0 release of futurist is not shown because that version indicates non-support for Python 2, and this is installing in a Python 2 environment, so pip doesn't even consider installing it.)

From that output, I can tell exactly what packages depend on futurist, and I can tell exactly what file and what line I need to modify.

I updated the Makefile and test-requirements.txt to use pip-compile instead of pip-conflict-checker since the output is so much more helpful.

Once I fix the dependency conflicts, the output of pip-compile is much better (this is only the relevant snippets of the full output):

# ...
futures==3.3.0            # via -r req.txt, apscheduler, futurist, openstacksdk, tenacity, tooz
futurist==1.10.0          # via -r req.txt, tooz
# ...
openstacksdk==0.43.0      # via -r req.txt, osc-lib
# ...
osc-lib==2.0.0            # via -r req.txt, python-mistralclient
# ...
git+https://github.com/StackStorm/python-mistralclient.git#egg=python-mistralclient  # via -r req.txt
# ...
tooz==1.66.1              # via -r req.txt
# ...
# The following packages are considered to be unsafe in a requirements file:
# setuptools

Now, this is a little weird, because openstacksdk version 0.43.0 only depends on the futures package, not the futurist package. But whatever, this fixes the build. And version 0.44.0 of openstacksdk only depends on the futurist package.

Lastly, I add a .git to the python-mistralclient dependency in st2actions/in-requirements.txt to ensure that the dependency only shows up once when combined, sorted, and uniquified with the other requirements.txt files in the Makefile.

@pull-request-size pull-request-size bot added the size/S PR that changes 10-29 lines. Very easy to review. label Mar 27, 2020
@blag blag added this to the 3.2.0 milestone Mar 27, 2020
Copy link
Contributor

@m4dcoder m4dcoder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@blag blag force-pushed the fix-futurist-version-conflict branch from 13895ba to fabe46b Compare March 27, 2020 19:40
@blag blag mentioned this pull request Apr 1, 2020
5 tasks
@blag
Copy link
Contributor Author

blag commented Apr 1, 2020

Updating to setuptools >= 42.0 in #4895 fixed this properly. Closing.

@blag blag closed this Apr 1, 2020
@blag blag deleted the fix-futurist-version-conflict branch April 1, 2020 17:44
@arm4b arm4b removed this from the 3.2.0 milestone Apr 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug size/S PR that changes 10-29 lines. Very easy to review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants