Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Instructions to create a development environment for M1/M2/osx-arm64 #8419

Closed
kafitzgerald opened this issue Nov 6, 2023 · 8 comments · Fixed by #8441
Closed

Instructions to create a development environment for M1/M2/osx-arm64 #8419

kafitzgerald opened this issue Nov 6, 2023 · 8 comments · Fixed by #8441
Labels
needs triage Issue that has not been reviewed by xarray team member topic-documentation

Comments

@kafitzgerald
Copy link
Contributor

What is your issue?

The current instructions to create a development environment for MacOS don't work for an M1/M2 Mac.

The conda environment solve fails on cdsm2 presumably because it's not available for osx-arm64.

I tried looking for related discussion and/or issues with recommendations, but didn't see any.
@kafitzgerald kafitzgerald added the needs triage Issue that has not been reviewed by xarray team member label Nov 6, 2023
@kafitzgerald
Copy link
Contributor Author

After removing cdsm2 from the environment yml file, installing, and running the test suite I'm seeing a bunch of failures related to pandas imports:
ImportError: Pandas requires version '2022.03.0' or newer of 'xarray' (version '0.1.dev4835+g31c6e62.d20231106' currently installed)

Happy to add more detail here and/or log a separate issue if that would be helpful.

@jhamman
Copy link
Member

jhamman commented Nov 6, 2023

@kafitzgerald - thanks for filing this issue. We should update the contributing docs here to make this an easier process.

For your second problem, can you share what xr.show_versions() returns? It seems like the test suite is picking up an old version of xarray?

@kafitzgerald
Copy link
Contributor Author

No problem (see below). I think it's just the editable install it's seeing?

versions >>> xr.show_versions() /Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit: None
python: 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:09:17) [Clang 16.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 21.6.0
machine: arm64
processor: arm
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.2

xarray: 0.1.dev4835+g31c6e62.d20231106
pandas: 2.1.2
numpy: 1.24.4
scipy: 1.11.3
netCDF4: 1.6.5
pydap: installed
h5netcdf: 1.2.0
h5py: 3.10.0
Nio: None
zarr: 2.16.1
cftime: 1.6.3
nc_time_axis: 1.4.1
PseudoNetCDF: 3.2.2
iris: 3.7.0
bottleneck: 1.3.7
dask: 2023.10.1
distributed: 2023.10.1
matplotlib: 3.8.1
cartopy: 0.22.0
seaborn: 0.13.0
numbagg: 0.6.0
fsspec: 2023.10.0
cupy: None
pint: 0.20.1
sparse: 0.14.0
flox: 0.8.1
numpy_groupies: 0.10.2
setuptools: 68.2.2
pip: 23.3.1
conda: None
pytest: 7.4.3
mypy: None
IPython: None
sphinx: None

@max-sixty
Copy link
Collaborator

That is surprising — we try and fallback to a much higher version if setuptools_scm resolve the version...

I don't have the g31c6e62 commit on my end locally — could we try switching to the main branch?

Not to get too deep into git, but are the tags updated?

@kafitzgerald
Copy link
Contributor Author

🤦🏻‍♀️ I hadn't fetched the tags. My bad (nothing wrong with the contrib docs here). All is looking good now on that front.

Interestingly I did see what looks to be a duplicate of #7879 consistently when not using the optional -n argument.

log for the segfault on test_open_mfdataset_manyfiles 
pytest xarray     
========================================================= test session starts =========================================================
platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/katelynw/xarray
configfile: pyproject.toml
plugins: timeout-2.2.0, cov-4.1.0, hypothesis-6.88.3, xdist-3.3.1, env-1.1.1
collected 17513 items / 1 skipped                                                                                                     

xarray/tests/test_accessor_dt.py .............................................................................................. [  0%]
............................................................................................................................... [  1%]
.................................................................                                                               [  1%]
xarray/tests/test_accessor_str.py ............................................................................................. [  2%]
............................................................................................................................... [  2%]
.......                                                                                                                         [  2%]
xarray/tests/test_array_api.py ...........                                                                                      [  2%]
xarray/tests/test_backends.py ..................................x................................................s............. [  3%]
......................x...............................................s.s................................x..................... [  4%]
....................................................................................................................x.......... [  4%]
............................................................................................................................... [  5%]
x...................................................................................................................sssssssssss [  6%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [  7%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [  7%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [  8%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [  9%]
ssssssssssssssssssssssssssssssssss...................................x........................................................x [ 10%]
......................ss..................................x............................................................x....... [ 10%]
..................................................x.......................................................x.................... [ 11%]
.......................................x....................................................................................... [ 12%]
...x..........................................................................................x................................ [ 12%]
.......................s.XXXXXXXXXXXXXXXXXXXXXXXXFatal Python error: Segmentation fault

Thread 0x000000017d3cf000 (most recent call first):
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x000000017c3c3000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 418 in open_store_variable
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 443 in <genexpr>
  File "/Users/katelynw/xarray/xarray/core/utils.py", line 471 in FrozenDict
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 442 in get_variables
  File "/Users/katelynw/xarray/xarray/backends/common.py", line 210 in load
  File "/Users/katelynw/xarray/xarray/backends/store.py", line 43 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 617 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000017b3b7000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 347 in __init__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 400 in open
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 603 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000017a3ab000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 348 in __init__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 400 in open
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 603 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000017939f000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 348 in __init__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 400 in open
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 603 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x0000000178393000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 447 in get_attrs
  File "/Users/katelynw/xarray/xarray/backends/common.py", line 212 in load
  File "/Users/katelynw/xarray/xarray/backends/store.py", line 43 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 617 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x0000000177387000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 68 in __setitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 221 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 347 in __init__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 400 in open
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 603 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000017637b000 (most recent call first):
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 372 in notify
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 484 in release
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x000000017536f000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 68 in __setitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 221 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 347 in __init__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 400 in open
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 603 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x0000000174363000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 447 in get_attrs
  File "/Users/katelynw/xarray/xarray/backends/common.py", line 212 in load
  File "/Users/katelynw/xarray/xarray/backends/store.py", line 43 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 617 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x0000000173357000 (most recent call first):
  File "/Users/katelynw/xarray/xarray/backends/lru_cache.py", line 55 in __getitem__
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 211 in _acquire_with_cache_info
  File "/Users/katelynw/xarray/xarray/backends/file_manager.py", line 199 in acquire_context
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 403 in _acquire
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 409 in ds
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 443 in get_variables
  File "/Users/katelynw/xarray/xarray/backends/common.py", line 210 in load
  File "/Users/katelynw/xarray/xarray/backends/store.py", line 43 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/netCDF4_.py", line 617 in open_dataset
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 573 in open_dataset
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 127 in _execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 225 in execute_task
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in <listcomp>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 239 in batch_execute_tasks
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x0000000102fc0580 (most recent call first):
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/threading.py", line 320 in wait
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/queue.py", line 171 in get
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 138 in queue_get
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 501 in get_async
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py", line 90 in get
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py", line 628 in compute
  File "/Users/katelynw/xarray/xarray/backends/api.py", line 1035 in open_mfdataset
  File "/Users/katelynw/xarray/xarray/tests/test_backends.py", line 3533 in test_open_mfdataset_manyfiles
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/Users/katelynw/miniconda3/envs/xarray-tests/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/Users/katelynw/miniconda3/envs/xarray-tests/bin/pytest", line 10 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, numexpr.interpreter, bottleneck.move, bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, _cffi_backend, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.linalg._flinalg, cftime._cftime, cytoolz.utils, cytoolz.itertoolz, cytoolz.functoolz, cytoolz.dicttoolz, cytoolz.recipes, xxhash._xxhash, psutil._psutil_osx, psutil._psutil_posix, markupsafe._speedups, numpy.linalg.lapack_lite, matplotlib._c_internal_utils, PIL._imaging, matplotlib._path, kiwisolver._cext, _brotli, netCDF4._netCDF4, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyproj._compat, pyproj._datadir, pyproj._network, pyproj._geod, pyproj.list, pyproj._crs, pyproj.database, pyproj._transformer, pyproj._sync, matplotlib._image, rasterio._version, rasterio._err, rasterio._filepath, rasterio._env, rasterio._transform, rasterio._base, rasterio.crs, rasterio._features, rasterio._warp, rasterio._io, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, msgpack._cmsgpack, numcodecs.jenkins, numcodecs.vlen, numcodecs.fletcher32, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, shapely.lib, shapely._geos, shapely._geometry_helpers, cartopy.trace, scipy.fftpack.convolve, tornado.speedups, cf_units._udunits2, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils (total: 250)
zsh: segmentation fault  pytest xarray

@max-sixty
Copy link
Collaborator

🤦🏻‍♀️ I hadn't fetched the tags. My bad (nothing wrong with the contrib docs here). All is looking good now on that front.

No stress!

Out of interest, was this an old clone? Or did a new clone hit this issue? We should fix on our end if a new clone could have this issue...

Interestingly I did see what looks to be a duplicate of #7879 consistently when not using the optional -n argument.

Interesting, thanks. These are plausibly bugs in netCDF. Xarray is pure-python, so we can't segfault ourselves, though we can call libraries incorrectly. It's somewhat of an intractable problem to push these upstream though...

In the meantime, if you don't need to run netcdf tests, uninstalling it will work!

@keewis
Copy link
Collaborator

keewis commented Nov 6, 2023 •
edited
Loading

just for reference, we're in the process of removing cdms2 support entirely from xarray (it is deprecated since the first release after May this year): cdms2 itself is in maintenance-only mode and will be archived end of this year. For the removal to happen in xarray, we just need to decide on a date / target version.

As for the segfaults: these are caused by netcdf4 dropping the global library lock in recent versions, which exposed our own locks not working properly for netcdf4 (but only for a threading scheduler). Concurrency issues like this are really tricky to debug, though, which I think is the reason why nobody got around to fixing these yet.

@kafitzgerald
Copy link
Contributor Author

Out of interest, was this an old clone? Or did a new clone hit this issue? We should fix on our end if a new clone could have this issue...

It was a new clone of my new fork, but I hadn't added or fetched the upstream repo yet per the instructions and therefore didn't have the tags.

And thanks for the info on cdms2 and netcdf4!

@keewis keewis mentioned this issue Nov 10, 2023
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs triage Issue that has not been reviewed by xarray team member topic-documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

remove cdms2 keewis/xarray
4 participants
@jhamman @max-sixty @kafitzgerald @keewis