Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure dev environment uses Arrow GPU packages #8637

Merged
merged 3 commits into from
Jul 2, 2021

Conversation

charlesbluca
Copy link
Member

With #7495 merged, it seems like the dev environment files create an environment with the CPU packages for pyarrow and arrow-cpp; this results in failure when trying to compile libcudf or import cudf:

ModuleNotFoundError: No module named 'pyarrow._cuda'
    from cudf import rmm
  File "/opt/conda/lib/python3.8/site-packages/cudf/__init__.py", line 11, in <module>
    from cudf import core, datasets, testing
  File "/opt/conda/lib/python3.8/site-packages/cudf/core/__init__.py", line 3, in <module>
    from cudf.core import _internals, buffer, column, column_accessor, common
  File "/opt/conda/lib/python3.8/site-packages/cudf/core/_internals/__init__.py", line 3, in <module>
    from cudf.core._internals.where import where
  File "/opt/conda/lib/python3.8/site-packages/cudf/core/_internals/where.py", line 11, in <module>
    from cudf.core.column import ColumnBase
  File "/opt/conda/lib/python3.8/site-packages/cudf/core/column/__init__.py", line 3, in <module>
    from cudf.core.column.categorical import CategoricalColumn
  File "/opt/conda/lib/python3.8/site-packages/cudf/core/column/categorical.py", line 25, in <module>
    from cudf import _lib as libcudf
  File "/opt/conda/lib/python3.8/site-packages/cudf/_lib/__init__.py", line 4, in <module>
    from . import (
ImportError: libarrow_cuda.so.400: cannot open shared object file: No such file or directory

This updates the dev environments and recipe to ensure that the GPU package of pyarrow (and arrow-cpp accordingly) are used.

@charlesbluca charlesbluca requested a review from a team as a code owner July 1, 2021 19:12
@github-actions github-actions bot added the conda label Jul 1, 2021
@charlesbluca charlesbluca added 3 - Ready for Review Ready for review by team bug Something isn't working non-breaking Non-breaking change labels Jul 1, 2021
Copy link
Member

@raydouglass raydouglass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charlesbluca
Copy link
Member Author

Do you know if the formatting for the recipe is correct? Not sure if the syntax differs significantly from the conda env file

@codecov
Copy link

codecov bot commented Jul 1, 2021

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@a0b0eab). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 9506fda differs from pull request most recent head 7feba9a. Consider uploading reports for the commit 7feba9a to get more accurate results
Impacted file tree graph

@@               Coverage Diff               @@
##             branch-21.08    #8637   +/-   ##
===============================================
  Coverage                ?   10.61%           
===============================================
  Files                   ?      109           
  Lines                   ?    18648           
  Branches                ?        0           
===============================================
  Hits                    ?     1980           
  Misses                  ?    16668           
  Partials                ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0b0eab...7feba9a. Read the comment docs.

@jakirkham
Copy link
Member

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 167c2b7 into rapidsai:branch-21.08 Jul 2, 2021
@jakirkham
Copy link
Member

Thanks all! 😄

@jakirkham
Copy link
Member

Looks like we needed another change to the recipe. Added here ( #8651 )

rapids-bot bot pushed a commit that referenced this pull request Jul 5, 2021
Follow up to PR ( #8637 )

Even though we pinned `*cuda` in `host`, this didn't carry over to the `run` dependencies. To fix this, tack on `*cuda` to `pyarrow` and `arrow-cpp` in `run`. This should fix installation errors where the CPU builds were getting pulled in unintentionally.

cc @ajschmidt8 @charlesbluca @pentschev

Authors:
  - https://github.com/jakirkham

Approvers:
  - Jordan Jacobelli (https://github.com/Ethyling)

URL: #8651
@charlesbluca charlesbluca deleted the fix-pyarrow-conda branch August 3, 2021 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team bug Something isn't working non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants