Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xarray - compatibility with latest versions #275

Open
johnomotani opened this issue Jan 19, 2023 · 9 comments · May be fixed by #276
Open

xarray - compatibility with latest versions #275

johnomotani opened this issue Jan 19, 2023 · 9 comments · May be fixed by #276
Labels
bug Something isn't working

Comments

@johnomotani
Copy link
Collaborator

johnomotani commented Jan 19, 2023 •
edited
Loading

Some change in xarray-2022.9.0 introduced a significant performance regression (see #213 (comment)). Possibly something related to the default option used to combine coordinates/metadata when combining two DataArrays/Datasets.

There also seem to be occasional errors when using the latest xarray versions - possibly some deprecated things that we need to update in xBOUT?

It is not good to not be able to use the latest versions of xarray. It would be really good if someone could work out what these issues are and how to fix them!

#234, #220, #213 are related.

The current workaround is to restrict xarray to versions <2022.9.0 #274.
@johnomotani johnomotani added the bug Something isn't working label Jan 19, 2023
@ZedThree
Copy link
Member

I can rustle up some RSE effort to look at this, but it might not be for a little while.

@johnomotani
Copy link
Collaborator Author

@ZedThree that would be awesome. I don't have time to look myself. It's not super urgent as the last good version (xarray-2022.6.0) is fine for now.

This was referenced Jan 20, 2023
Open
Draft
@johnomotani
Copy link
Collaborator Author

Would be interesting to see if the performance regression is fixed by the latest xarray release - pydata/xarray#7824 may help things.

@dschwoerer
Copy link
Contributor

Output of pytest with xarray 2023.7.0 and python 3.12:

+ /usr/bin/pytest xbout --long
============================= test session starts ==============================
platform linux -- Python 3.12.0b4, pytest-7.3.2, pluggy-1.0.0
rootdir: /builddir/build/BUILD/xbout-0.3.5
collected 796 items
xbout/calc/tests/test_turbulence.py .....                                [  0%]
xbout/tests/test_against_collect.py ........                             [  1%]
xbout/tests/test_animate.py .......................s...............      [  6%]
xbout/tests/test_boutdataarray.py ...................................... [ 11%]
..................                                                       [ 13%]
xbout/tests/test_boutdataset.py ........................................ [ 18%]
..........................ss...................                          [ 24%]
xbout/tests/test_fastoutput.py .                                         [ 24%]
xbout/tests/test_geometries.py ..                                        [ 24%]
xbout/tests/test_grid.py .....                                           [ 25%]
xbout/tests/test_init.py .                                               [ 25%]
xbout/tests/test_load.py ......................................s........ [ 31%]
.s...................................................................... [ 40%]
............                                                             [ 42%]
xbout/tests/test_plot.py ............................................... [ 47%]
........................................................................ [ 57%]
.........................                                                [ 60%]
xbout/tests/test_region.py ............................................. [ 65%]
........................................................................ [ 74%]
........................................................................ [ 83%]
........................................................................ [ 92%]
...........................................                              [ 98%]
xbout/tests/test_utils.py .............                                  [100%]
=============================== warnings summary ===============================
../../../../usr/lib/python3.12/site-packages/pytz/tzinfo.py:27
  /usr/lib/python3.12/site-packages/pytz/tzinfo.py:27: DeprecationWarning: datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.fromtimestamp(timestamp, datetime.UTC).
    _epoch = datetime.utcfromtimestamp(0)
xbout/tests/test_animate.py::TestAnimate::test_animate2D_controls_arg[controls4]
xbout/tests/test_animate.py::TestAnimate::test_animate1D
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls3]
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls4]
xbout/tests/test_animate.py::TestAnimate::test_animate1D_controls_arg[controls5]
xbout/tests/test_animate.py::TestAnimate::test_animate_list
xbout/tests/test_animate.py::TestAnimate::test_animate_list_controls_arg[controls3]
xbout/tests/test_animate.py::TestAnimate::test_animate_list_controls_arg[controls4]
xbout/tests/test_boutdataarray.py::TestBoutDataArrayMethods::test_to_dataset
  /usr/lib64/python3.12/site-packages/matplotlib/animation.py:884: UserWarning: Animation was deleted without rendering anything. This is most likely not intended. To prevent deletion, assign the Animation to a variable, e.g. `anim`, that exists until you output the Animation using `plt.show()` or `anim.save()`.
    warnings.warn(
xbout/tests/test_boutdataarray.py: 32 warnings
xbout/tests/test_boutdataset.py: 22 warnings
  /builddir/build/BUILD/xbout-0.3.5/xbout/geometries.py:390: UserWarning: rename 't' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename(t=coordinates["t"])
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_slab
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_CENTRE]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_XLOW]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_YLOW]
xbout/tests/test_boutdataset.py::TestBoutDatasetMethods::test_integrate_midpoints_salpha[CELL_ZLOW]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
xbout/tests/test_load.py::TestOpen::test_restarts
  /builddir/build/BUILD/xbout-0.3.5/xbout/load.py:349: UserWarning: No geometry type found, no physical coordinates will be added
    warn("No geometry type found, no physical coordinates will be added")
xbout/tests/test_boutdataset.py::TestSave::test_reload_all[toroidal]
  /builddir/build/BUILD/xbout-0.3.5/xbout/load.py:245: UserWarning: open_boutdataset() called with geometry="toroidal", but we are reloading a Dataset that was saved after being loaded with geometry="toroidal". Applying geometry="toroidal" from the argument.
    warn(
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 't' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'x' to 'x' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'y' to 'y' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[None]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart[1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes0]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_to_restart_change_npe_doublenull_expect_fail[npes1]
xbout/tests/test_boutdataset.py::TestSaveRestart::test_from_restart_to_restart
  /builddir/build/BUILD/xbout-0.3.5/xbout/utils.py:376: UserWarning: rename 'z' to 'z' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename({ds.metadata[key]: d})
xbout/tests/test_load.py::TestOpen::test_single_file
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:838: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_squashed_file
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:864: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_x
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:978: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_y
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:1012: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-False-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-True-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths0-True-True]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-False-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-True-False]
xbout/tests/test_load.py::TestOpen::test_combine_along_xy[lengths1-True-True]
  /builddir/build/BUILD/xbout-0.3.5/xbout/tests/test_load.py:1076: UserWarning: rename 't_array' to 't' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    expected = expected.set_coords(["t_array", "dx", "dy", "dz"]).rename(
xbout/tests/test_load.py::TestOpen::test_salpha
xbout/tests/test_load.py::TestOpen::test_salpha
  /builddir/build/BUILD/xbout-0.3.5/xbout/geometries.py:483: UserWarning: rename 'x' to 'r' does not create an index anymore. Try using swap_dims instead or use set_index after rename to create an indexed coordinate.
    ds = ds.rename(x="r")
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 791 passed, 5 skipped, 111 warnings in 18885.95s (5:14:45) =========

Good thing is no errors. And there is also some potential to improve performance ;-)

@dschwoerer
Copy link
Contributor

https://src.fedoraproject.org/rpms/python-xbout/blob/rawhide/f/python-xbout.spec#_13

This was done with #278 #283 #291

@mrhardman
Copy link
Collaborator

mrhardman commented Nov 4, 2024 •
edited
Loading

I have had problems with this issue for a new installation of a python venv using xbout. I had a conflict between numpy and xbout because xbout 0.3.6 requires xarray<2022.9.0,>=0.18.0, but you have xarray 2024.10.0 which is incompatible. It would be very good to be able to use the latest xarray and numpy versions! The conflict was resolved by installing xarray==2022.6.0 and numpy==1.26.4.

@bendudson
Copy link
Contributor

bendudson commented Nov 4, 2024 •
edited
Loading

Hi @mrhardman ! This should be fixed by this PR #303 but there is something odd going on with python 3.12 and 3.13: They segfault or hang on the Github CI somewhere inside Dask and Numpy. Python 3.13 works fine locally. Which version of python are you using?

@mrhardman
Copy link
Collaborator

The version was python 3.13.0. I can try to reproduce the error with a fresh venv if it is useful?

@dschwoerer
Copy link
Contributor

I also see random segfaults during the tests on fedora. I have not managed to figure out what is going on.
They are only randomly appearing, so maybe valgrind could be used to check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Compatibility with xarray>=2022.9.0 boutproject/xBOUT
5 participants
@bendudson @ZedThree @johnomotani @dschwoerer @mrhardman