Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extras not translated to rez #66

Open
brycegbrazen opened this issue Nov 13, 2023 · 9 comments
Open

Extras not translated to rez #66

brycegbrazen opened this issue Nov 13, 2023 · 9 comments
Labels
enhancement New feature or request

Comments

@brycegbrazen
Copy link

When running a rez-pip2 on the boxsdk PyPi package with the extra "jwt" specified, the extra dependencies are not included in the resulting package.py file.

Environment

  • Windows 10
  • Rez version 2.112.0
  • Rez python version 3.7
  • rez-pip 0.4.0.dev1

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7

Expected behavior
"jwt" and "cryptography" packages listed in "extra_requires" on boxsdk's setup.py file are included in requires portion of resulting package.py.

Actual behavior
"jwt" and "cryptography" packages are missing from the resulting requires portion of the package.py.
@JeanChristopheMorinPerso
Copy link
Owner

Hi @brycegbrazen , can you provide me with the full output logs please? Also, could you try with rez-pip2 'boxsdk[jwt]==3.9.2' --python-version 3.7 (quoting boxsdk[jwt]==3.9.2)? I remember seeing something like this while I was working on the early PoC...

@brycegbrazen
Copy link
Author

@JeanChristopheMorinPerso Does it actually log to a file? Not seeing anything in the docs that mention that so I figured I'd ask. Otherwise I can just upload the output from my stdout.

@JeanChristopheMorinPerso
Copy link
Owner

No, it logs everything to stdout/stderr.

@brycegbrazen
Copy link
Author

Looks like it includes those extra dependencies, but just doesn't include them as requirements in the package.py. I'll try quoting the package request next.

(Rez) λ rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7
INFO     Installing requested packages for Python 3.7.9
INFO     Resolved 17 dependencies for python 3.7.9
INFO     Downloading...
INFO     Downloaded PyJWT-2.8.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\PyJWT-2.8.0-py3-none-any.whl' (22591
         bytes)
INFO     Downloaded idna-3.4 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\idna-3.4-py3-none-any.whl' (61538 bytes)
INFO     Downloaded boxsdk-3.9.2 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\boxsdk-3.9.2-py2.py3-none-any.whl'
         (139209 bytes)
INFO     Downloaded typing_extensions-4.7.1 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\typing_extensions-4.7.1-py3-none-any.whl' (33232 bytes)
INFO     Downloaded requests-2.31.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests-2.31.0-py3-none-any.whl'
         (62574 bytes)
INFO     Downloaded urllib3-2.0.7 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\urllib3-2.0.7-py3-none-any.whl'
         (124213 bytes)
INFO     Downloaded python-dateutil-2.8.2 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\python_dateutil-2.8.2-py2.py3-none-any.whl' (247702 bytes)
INFO     Downloaded charset-normalizer-3.3.2 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\charset_normalizer-3.3.2-cp37-cp37m-win_amd64.whl' (98127 bytes)
INFO     Downloaded attrs-23.1.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\attrs-23.1.0-py3-none-any.whl' (61160
         bytes)
INFO     Downloaded requests-toolbelt-1.0.0 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests_toolbelt-1.0.0-py2.py3-none-any.whl' (54481 bytes)
INFO     Downloaded six-1.16.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\six-1.16.0-py2.py3-none-any.whl' (11053
         bytes)
INFO     Downloaded importlib-metadata-6.7.0 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\importlib_metadata-6.7.0-py3-none-any.whl' (22934 bytes)
INFO     Downloaded zipp-3.15.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\zipp-3.15.0-py3-none-any.whl' (6758
         bytes)
INFO     Downloaded certifi-2023.7.22 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\certifi-2023.7.22-py3-none-any.whl' (158334 bytes)
INFO     Downloaded cffi-1.15.1 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cffi-1.15.1-cp37-cp37m-win_amd64.whl'
         (179301 bytes)
INFO     Downloaded pycparser-2.21 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\pycparser-2.21-py2.py3-none-any.whl'
         (118697 bytes)
INFO     Downloaded cryptography-41.0.5 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cryptography-41.0.5-cp37-abi3-win_amd64.whl' (2669522 bytes)
INFO     Downloaded 17 wheels
INFO     Installing boxsdk-3.9.2 wheel
INFO     Installing attrs-23.1.0 wheel
INFO     Installing cryptography-41.0.5 wheel
INFO     Installing PyJWT-2.8.0 wheel
INFO     Installing requests-2.31.0 wheel
INFO     Installing requests-toolbelt-1.0.0 wheel
INFO     Installing urllib3-2.0.7 wheel
INFO     Installing python-dateutil-2.8.2 wheel
INFO     Installing certifi-2023.7.22 wheel
INFO     Installing cffi-1.15.1 wheel
INFO     Installing charset-normalizer-3.3.2 wheel
INFO     Installing idna-3.4 wheel
INFO     Installing six-1.16.0 wheel
INFO     Installing importlib-metadata-6.7.0 wheel
INFO     Installing typing_extensions-4.7.1 wheel
INFO     Installing zipp-3.15.0 wheel
INFO     Installing pycparser-2.21 wheel
INFO     Creating rez package for boxsdk
INFO     Installing boxsdk-3.9.2 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for attrs
INFO     Installing attrs-23.1.0 [importlib_metadata]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for cryptography
INFO     Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for PyJWT
INFO     Installing PyJWT-2.8.0 [typing_extensions]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for requests
INFO     Installing requests-2.31.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for requests-toolbelt
INFO     Installing requests_toolbelt-1.0.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for urllib3
INFO     Installing urllib3-2.0.7 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for python-dateutil
INFO     Installing python_dateutil-2.8.2 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for certifi
INFO     Installing certifi-2023.7.22 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for cffi
INFO     Installing cffi-1.15.1 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for charset-normalizer
INFO     Installing charset_normalizer-3.3.2 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for idna
INFO     Installing idna-3.4 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for six
INFO     Installing six-1.16.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for importlib-metadata
INFO     Installing importlib_metadata-6.7.0 [typing_extensions-3.6.4+]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for typing_extensions
INFO     Installing typing_extensions-4.7.1 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for zipp
INFO     Installing zipp-3.15.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for pycparser
INFO     Installing pycparser-2.21 []
INFO     Created 1 variants and skipped 0

@JeanChristopheMorinPerso
Copy link
Owner

JeanChristopheMorinPerso commented Nov 13, 2023 •
edited
Loading

I just tried on my side and it's working. And from what I see in your log, it seems like it also worked for you. The extra is called jwt and it installs PyJWT and cryptography (https://github.com/box/box-python-sdk/blob/v3.9.2/setup.py#L63)

@JeanChristopheMorinPerso
Copy link
Owner

I see

INFO     Creating rez package for cryptography
INFO     Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for PyJWT
INFO     Installing PyJWT-2.8.0 [typing_extensions]
INFO     Created 1 variants and skipped 0

Which means it created a package cryptography-41.0.5 and PyJWT-2.8.0.

@brycegbrazen
Copy link
Author

Yeah I see it's installing those two extra dependencies, but it doesn't actually include them in the resolve when I run a
rez-env boxsdk. I thought it'd include PyJWT and cryptography as dependencies in the package.py somehow.

@JeanChristopheMorinPerso
Copy link
Owner

Ah yeah. Well, the boxsdk package will be boxsdk only... We don't try to merge things (as it would be had to know what you really want to do. If you pip install boxsdk[jwt], you will endup with everything on one env, but it doesn't modify dependencies. In other words

$ python
>>> import importlib.metadata
>>> dist = importlib.metadata.distribution('boxsdk') 
>>> dist.requires
['attrs (>=17.3.0)', 'urllib3', 'requests (<3,>=2.4.3)', 'requests-toolbelt (>=0.4.0)', 'python-dateutil', "coveralls ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'dev'", "tox-gh-actions ; extra == 'gh'", "tox (<=3.28.0) ; extra == 'gh'", "pyjwt (>=1.7.0) ; extra == 'jwt'", "cryptography (>=3) ; extra == 'jwt'", "redis (>=2.10.3) ; extra == 'redis'", "bottle ; extra == 'test'", "jsonpatch (>1.14) ; extra == 'test'", "sqlalchemy (<1.4.0) ; extra == 'test'", "pytest ; extra == 'test'", "pytest-timeout ; extra == 'test'", "pytest-cov ; extra == 'test'", "pytest-lazy-fixture ; extra == 'test'", "pytz ; extra == 'test'", "urllib3 (<2) ; extra == 'test'"]

The dependencies aren't merged. The xtras are still extras even after the install.

rez-pip does the same (both the builtin rez-pip and rez-pip2).

We additionally don't have a good way to translate boxsdk[jwt] to rez-env boxsdk[jwt]. See

rez-pip/src/rez_pip/utils.py

Lines 490 to 493 in 0217569

TODO: Currently there is no way to reflect extras that may have been chosen
for this pip package. We need to wait for rez "package features" before this
will be possible. You probably shouldn't use extras presently.
. We simply don't have a way to do that right now.

@JeanChristopheMorinPerso
Copy link
Owner

Maybe ephemerals could help, but it would need some investigation to see if they would work or not.

@JeanChristopheMorinPerso JeanChristopheMorinPerso changed the title Specified extras not included in pip converted package Extras not translated to rez Nov 14, 2023
@JeanChristopheMorinPerso JeanChristopheMorinPerso added the enhancement New feature or request label Nov 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JeanChristopheMorinPerso @brycegbrazen