Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] cuDF-python import-time error with pyarrow #13679

Closed
GregoryKimball opened this issue Jul 10, 2023 · 3 comments
Closed

[BUG] cuDF-python import-time error with pyarrow #13679

GregoryKimball opened this issue Jul 10, 2023 · 3 comments
Labels
1 - On Deck To be worked on next bug Something isn't working CMake CMake build issue Python Affects Python cuDF API.

Comments

@GregoryKimball
Copy link
Contributor

GregoryKimball commented Jul 10, 2023

Describe the bug
@benfred reported an import-time error with cuDF-python

In [13]: import cudf
df <jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
---------------------------------------------------------------------------
ArrowMemoryError                          Traceback (most recent call last)
<ipython-input-13-e13365c50bc4> in <module>
----> 1 import cudf

/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/__init__.py in <module>
     74 from cudf.core.tools.datetimes import DateOffset, date_range, to_datetime
     75 from cudf.core.tools.numeric import to_numeric
---> 76 from cudf.io import (
     77     from_dlpack,
     78     read_avro,

/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/io/__init__.py in <module>
      7 from cudf.io.json import read_json
      8 from cudf.io.orc import read_orc, read_orc_metadata, to_orc
----> 9 from cudf.io.parquet import (
     10     ParquetDatasetWriter,
     11     merge_parquet_filemetadata,

/opt/conda/envs/rapids/lib/python3.10/site-packages/cudf/io/parquet.py in <module>
     15 import numpy as np
     16 import pandas as pd
---> 17 from pyarrow import dataset as ds, parquet as pq
     18 
     19 import cudf

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/dataset.py in <module>
     21 from pyarrow.util import _is_iterable, _stringify_path, _is_path_like
     22 
---> 23 from pyarrow._dataset import (  # noqa
     24     CsvFileFormat,
     25     CsvFragmentScanOptions,

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/_dataset.pyx in init pyarrow._dataset()

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/_compute.pyx in pyarrow._compute.Expression._scalar()

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/scalar.pxi in pyarrow.lib.scalar()

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/opt/conda/envs/rapids/lib/python3.10/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowMemoryError: malloc of size 64 failed

Steps/Code to reproduce bug
Build cudf from source, and then attempt to import cudf in python

Expected behavior
import succeeds

Environment overview (please complete the following information)

  • Environment location: docker
    image: rapidsai/rapidsai-core-arm64:23.06-cuda11.8-runtime-ubuntu22.04-py3.10
  • Method of cuDF install: from source
@GregoryKimball GregoryKimball added bug Something isn't working 1 - On Deck To be worked on next Python Affects Python cuDF API. CMake CMake build issue labels Jul 10, 2023
@vyasr
Copy link
Contributor

vyasr commented Jul 13, 2023

I believe this will be fixed by conda-forge/arrow-cpp-feedstock#1116, correct?

@GregoryKimball GregoryKimball changed the title [BUG] cuDF-python import-time error with pyarrow on Grace-Hopper [BUG] cuDF-python import-time error with pyarrow Jul 14, 2023
@benfred
Copy link
Member

benfred commented Jul 14, 2023

This is fixed with the changes to the arrow-cpp-feedstock -

@vyasr
Copy link
Contributor

vyasr commented Jul 14, 2023

Thanks!

@vyasr vyasr closed this as completed Jul 14, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in cuDF/Dask/Numba/UCX Jul 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1 - On Deck To be worked on next bug Something isn't working CMake CMake build issue Python Affects Python cuDF API.
Projects
Archived in project
Development

No branches or pull requests

3 participants