Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework python-cmethods: replace the CMethods class by the new adjust function #48

Merged
merged 40 commits into from
Jan 23, 2024

Conversation

btschwertfeger
Copy link
Owner

@btschwertfeger btschwertfeger commented Nov 3, 2023

To improve the performance of applied bias corrections, the complete CMethods class was removed. Individual techniques can't be executed directly by calling their function anymore. Also the adjust_3d function was removed.

Instead, a new function named adjust was added which acts like the old adjust_3d function but can handle single and multidimensional data sets and will return the bias corrected data as xarray.Dataset. By using xarrays ufunc function instead of iterating over entries per dimension, the execution time could be reduced drastically.

  • Since functions like linear_scaling, variance_scaling and so on are now internal functions and should not be called directly, they are not part of the documentation anymore. Instead, a new section in the documentation was added - which describes the implemented methods and provides some examples.

More changes:

  • split init.py into different modules
  • reworked the unit tests
  • updated examples
  • updated README
  • updated documentation
  • chunked data can now be adjusted more efficiently

Follow-up: Create issue for improving the DQM function (#48 (comment)).

@btschwertfeger btschwertfeger changed the title TBD Improve adjustments by using xarray's ufunc method Nov 12, 2023
@btschwertfeger btschwertfeger added the Breaking Breaking changes label Nov 12, 2023
Copy link

codecov bot commented Nov 12, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (9878005) 98.23% compared to head (e1868d8) 98.08%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #48      +/-   ##
==========================================
- Coverage   98.23%   98.08%   -0.15%     
==========================================
  Files           1        7       +6     
  Lines         283      261      -22     
==========================================
- Hits          278      256      -22     
  Misses          5        5              
Flag Coverage Δ
unittests 98.08% <98.03%> (-0.15%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
cmethods/__init__.py 100.00% <100.00%> (+1.76%) ⬆️
cmethods/scaling.py 100.00% <100.00%> (ø)
cmethods/static.py 100.00% <100.00%> (ø)
cmethods/types.py 100.00% <100.00%> (ø)
cmethods/core.py 97.22% <97.22%> (ø)
cmethods/distribution.py 98.00% <98.00%> (ø)
cmethods/utils.py 96.55% <96.55%> (ø)

@btschwertfeger btschwertfeger changed the title Improve adjustments by using xarray's ufunc method Improve performance by using xarray's ufunc function + drop Python < 3.11 support Nov 12, 2023
@btschwertfeger btschwertfeger marked this pull request as ready for review January 14, 2024 12:33
Copy link

@riley-brady riley-brady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job with this! I just focused mainly on the apply ufunc implementation. Just a few comments here. It worked on my example data over 1-4D, and very quickly with dask.

.github/workflows/release.yml Outdated Show resolved Hide resolved
cmethods/__init__.py Outdated Show resolved Hide resolved
cmethods/__init__.py Outdated Show resolved Hide resolved
cmethods/__init__.py Outdated Show resolved Hide resolved
cmethods/__init__.py Show resolved Hide resolved
cmethods/__init__.py Outdated Show resolved Hide resolved
cmethods/__init__.py Outdated Show resolved Hide resolved
@btschwertfeger btschwertfeger changed the title Improve performance by using xarray's ufunc function + drop Python < 3.11 support Improve performance by using xarray's ufunc function Jan 20, 2024
@btschwertfeger btschwertfeger changed the title Improve performance by using xarray's ufunc function Rework python-cmethods: replace the CMethods class with the new adjust function Jan 20, 2024
@btschwertfeger btschwertfeger changed the title Rework python-cmethods: replace the CMethods class with the new adjust function Rework python-cmethods: replace the CMethods class by the new adjust function Jan 20, 2024
Copy link

@riley-brady riley-brady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alles gute bei mir! Well done.

cmethods/utils.py Outdated Show resolved Hide resolved
@btschwertfeger btschwertfeger merged commit 222e929 into master Jan 23, 2024
35 checks passed
@btschwertfeger btschwertfeger deleted the apply-ufunc-corrections branch January 23, 2024 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Breaking Breaking changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimization for adjust_3d Find a solution to process large data sets more efficient
2 participants