pipenv won't install airflow #2583

maxgrenderjones · 2018-07-15T00:31:50Z

Issue description

When attempting to install airflow, pipenv refuses to create a lock file, even though (I think) there's not actually a problem

> mkdir airflow-pipenv
> cd airflow-pipenv
> pipenv install airflow[mysql]==1.8.0
Creating a virtualenv for this project...
Pipfile: /Users/mgrenderjones/Documents/workspace/airflow-pipenv/Pipfile
... snip snip...
Adding airflow[mysql]==1.8.0 to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches jinja2<2.9.0,==2.8.1,>=2.10,>=2.4,>=2.7.3,>=2.8
Tried: 2.0, 2.1, 2.1.1, 2.2, 2.2.1, 2.3, 2.3.1, 2.4, 2.4.1, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.8, 2.8, 2.8.1, 2.8.1, 2.9, 2.9, 2.9.1, 2.9.1, 2.9.2, 2.9.2, 2.9.3, 2.9.3, 2.9.4, 2.9.4, 2.9.5, 2.9.5, 2.9.6, 2.9.6, 2.10, 2.10

But it's not clear to me that there's a >=2.10.0 dependency:

> pipenv graph | grep -i jinja
    - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
  - jinja2 [required: >=2.7.3,<2.9.0, installed: 2.8.1]
    - Jinja2 [required: >=2.8, installed: 2.8.1]

Expected result

Airflow should install (with Jinja2==2.8.1)

Actual result

Seems I can't have airflow and a lock file.

Looking at the output of pipenv lock --verbose (and setup.py) the issue appears to be that airflow requires a restricted set of versions of Jinja2 (<=2.9.0) as well as specific versions of flask and it's helpers e.g. flask-[admin|swagger] (there are more), which themselves require >=2.10 of Jinja2. And so kaboom.

But my project doesn't use flask-admin etc, and I've been happily using airflow for a long time now without issue - is there no way to force the issue? Sure, we could say this is an 'upstream' problem, but it feels like there ought to be a way to say 'please let me do this'!

Steps to replicate

See above

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/Users/mgrenderjones/Library/Python/2.7/lib/python/site-packages/pipenv'

Python location: '/usr/local/opt/python@2/bin/python2.7'

Other Python installations in PATH:

2.7: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p/bin/python2.7
2.7: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p/bin/python2.7
2.7: /usr/local/bin/python2.7
2.7: /usr/bin/python2.7
2.7.15: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p/bin/python
2.7.15: /usr/local/bin/python
2.7.10: /usr/bin/python
2.7.15: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p/bin/python2
2.7.15: /usr/local/bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST 2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '2.7.15',
 'python_version': '2.7',
 'sys_platform': 'darwin'}

System environment variables:

_OLD_VIRTUAL_PATH
SPARK_HOME
PYTHONDONTWRITEBYTECODE
LOGNAME
USER
HOME
PATH
TERM_PROGRAM
LANG
TERM
Apple_PubSub_Socket_Render
SHLVL
XPC_FLAGS
_OLD_FISH_PROMPT_OVERRIDE
PIPENV_ACTIVE
TERM_SESSION_ID
XPC_SERVICE_NAME
SSH_AUTH_SOCK
VIRTUAL_ENV
SHELL
TERM_PROGRAM_VERSION
PIP_PYTHON_PATH
TMPDIR
__CF_USER_TEXT_ENCODING
PWD

Pipenv–specific environment variables:

PIPENV_ACTIVE: 1

Debug–specific environment variables:

PATH: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p/bin:/Users/mgrenderjones/.cargo/bin:/Users/mgrenderjones/Library/Python/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki
SHELL: /usr/bin/fish
LANG: en_US.UTF-8
PWD: /Users/mgrenderjones/Documents/workspace/airflow-pipenv
VIRTUAL_ENV: /Users/mgrenderjones/.local/share/virtualenvs/vulcan-pipenv-vMW1Zc4p

Contents of Pipfile ('/Users/mgrenderjones/Documents/workspace/airflow-pipenv/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
airflow = {version = "==1.8.0", extras = ["mysql"]}

[requires]
python_version = "2.7"

The text was updated successfully, but these errors were encountered:

maxgrenderjones · 2018-07-15T00:37:01Z

The solution feels related to #1921, but I can't see what the dev-requires workaround would be :(

uranusjr · 2018-07-15T06:22:36Z

The work around is to add jinja2 = "==2.9.0" explicitly in you Pipfile, this helps the resolver to get the correct version.

maxgrenderjones · 2018-07-15T20:42:25Z

Unless I'm holding it wrong, I don't think that works:

> mkdir pipenv-test
> cd pipenv-test
> pipenv shell
> pipenv install airflow[mysql] Jinja2==2.8.1
... snip snip ...
Could not find a version that matches jinja2<2.9.0,==2.8.1,>=2.10,>=2.4,>=2.7.3,>=2.8 (from -r /var/folders/q9/f3z9clp117dc1t0860v0xj9hx5rgx5/T/pipenv-ii8_Hs-requirements/pipenv-SFquyt-constraints.txt (line 3))
Tried: 2.0, 2.1, 2.1.1, 2.2, 2.2.1, 2.3, 2.3.1, 2.4, 2.4.1, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.8, 2.8, 2.8.1, 2.8.1, 2.9, 2.9, 2.9.1, 2.9.1, 2.9.2, 2.9.2, 2.9.3, 2.9.3, 2.9.4, 2.9.4, 2.9.5, 2.9.5, 2.9.6, 2.9.6, 2.10, 2.10

It's in the Pipfile:

> cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
airflow = {version = "*", extras = ["mysql"]}
"jinja2" = "==2.8.1"

[requires]
python_version = "2.7"

maxgrenderjones · 2018-07-15T22:02:59Z

Investigating the output of pipenv lock --verbose I get lines like:

flask-wtf==0.12 requires click>=5.1, flask, flask-wtf==0.12, itsdangerous>=0.24, jinja2>=2.10, markupsafe>=0.23, werkzeug, wtforms

At which point you might think it reasonable that we end up abandoning, since airflow requires Jinja2<2.9.0. But here's the thing - setup.py for flask-wtf-0.12 requires no such thing - install_requires only includes only 'Flask', 'Werkzeug', 'WTForms', and they don't require Jinja2>=2.10. So what gives? Is the solver pulling in the wrong versions somehow?

This suspicion is only heightened by the results after running pipenv install --skip-lock airflow - there doesn't appear to be any unsatisfiable condition:

> pipenv graph | grep -i jinja
    - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
      - Jinja2 [required: >=2.4, installed: 2.8.1]
  - jinja2 [required: >=2.7.3,<2.9.0, installed: 2.8.1]

Note that the most recent version, flask-1.0.2 does require Jinja2>=2.10. But even when you pin flask==0.11.1 it still seems to make the same mistake

JacobHayes · 2018-07-16T20:55:44Z

@maxgrenderjones Side note: you probably want the apache-airflow package. airflow is the old 1.8 version while apache-airflow is 1.8+. Source

maxgrenderjones · 2018-07-17T07:45:40Z

Thanks - am trying to replicate our current environment where we use a version from the `airflow` era. But regardless, `apache-airflow` doesn’t install either :(

…

On 16 Jul 2018, at 21:55, Jacob Hayes ***@***.***> wrote: @maxgrenderjones Side note: you probably want the apache-airflow package. airflow is the old 1.8 version while apache-airflow is 1.8+. Source — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

vkahawala · 2018-07-18T09:41:13Z

Almost same thing happened to me when trying to install prospector with pipenv.
I raised a stack overflow question as well.

Seems like pipenv is adding a additional restriction

pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]

Flask = "*"

[dev-packages]
prospector = "*"

Log for pipenv install --dev

Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches astroid<2.0,>=1.4,>=1.6,>=2.0.0
Tried: 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.0, 1.3.1, 1.3.1, 1.3.2, 1.3.2, 1.3.3, 1.3.3, 1.3.4, 1.3.4, 1.3.5, 1.3.5, 1.3.6, 1.3.6, 1.3.7, 1.3.7, 1.3.8, 1.3.8, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.4.2, 1.4.2, 1.4.3, 1.4.3, 1.4.4, 1.4.4, 1.4.5, 1.4.5, 1.4.6, 1.4.6, 1.4.7, 1.4.7, 1.4.8, 1.4.8, 1.4.9, 1.4.9, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.3, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 2.0, 2.0
Skipped pre-versions: 2.0.0.dev0, 2.0.0.dev0, 2.0.0.dev1, 2.0.0.dev1, 2.0.0.dev2, 2.0.0.dev2, 2.0.0.dev3, 2.0.0.dev3, 2.0.0.dev4, 2.0.0.dev4
There are incompatible versions in the resolved dependencies.

Astroid version that pipenv is looking for is restricted by this

Could not find a version that matches astroid<2.0,>=1.4,>=1.6,>=2.0.0

But when I installed using --skip-lock and check the graph, >=2.0.0 restriction is nowhere to be found.

- astroid [required: >=1.6,<2.0, installed: 1.6.5]
  - astroid [required: >=1.6,<2.0, installed: 1.6.5]
    - astroid [required: >=1.6,<2.0, installed: 1.6.5]
  - astroid [required: >=1.6,<2.0, installed: 1.6.5]
- astroid [required: >=1.4, installed: 1.6.5]

I'm not sure whether this has something to do with pre-versions. But appreciate any helpful thought

uranusjr · 2018-07-18T10:35:18Z

I’m closing this to guide people to #2596. That thread contains much more information, and should be much more useful to everyone.

arocketman · 2020-01-24T18:39:46Z

hi everyone, this is kind of unrelated but since the google search for "Pipenv airflow fails" brings to here I think I will also write that I solved my (dumb) problem by making sure I was doing

pipenv install apache-airflow

and not...

pipenv install airflow

:)

orf mentioned this issue Jul 17, 2018

Pipenv fails to resolve subdependency when viable solution exists #2596

Closed

uranusjr closed this as completed Jul 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipenv won't install airflow #2583

pipenv won't install airflow #2583

maxgrenderjones commented Jul 15, 2018 •

edited

Loading

maxgrenderjones commented Jul 15, 2018 •

edited

Loading

uranusjr commented Jul 15, 2018

maxgrenderjones commented Jul 15, 2018

maxgrenderjones commented Jul 15, 2018 •

edited

Loading

JacobHayes commented Jul 16, 2018

maxgrenderjones commented Jul 17, 2018 via email

vkahawala commented Jul 18, 2018

uranusjr commented Jul 18, 2018

arocketman commented Jan 24, 2020

pipenv won't install airflow #2583

pipenv won't install airflow #2583

Comments

maxgrenderjones commented Jul 15, 2018 • edited Loading

Issue description

Expected result

Actual result

Steps to replicate

maxgrenderjones commented Jul 15, 2018 • edited Loading

uranusjr commented Jul 15, 2018

maxgrenderjones commented Jul 15, 2018

maxgrenderjones commented Jul 15, 2018 • edited Loading

JacobHayes commented Jul 16, 2018

maxgrenderjones commented Jul 17, 2018 via email

vkahawala commented Jul 18, 2018

uranusjr commented Jul 18, 2018

arocketman commented Jan 24, 2020

maxgrenderjones commented Jul 15, 2018 •

edited

Loading

maxgrenderjones commented Jul 15, 2018 •

edited

Loading

maxgrenderjones commented Jul 15, 2018 •

edited

Loading