Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unyt_dask_array limitations with reduction methods #269

Open
chrishavlin opened this issue Sep 8, 2022 · 0 comments
Open

unyt_dask_array limitations with reduction methods #269

chrishavlin opened this issue Sep 8, 2022 · 0 comments

Comments

@chrishavlin
Copy link
Contributor

Opening this issue to track an issue with the initial implementation of unyt-dask arrays (which is approaching complete, #185).

The problem is that the unyt_dask_array class has no way of catching dask.array reductions. Operations like dask.array.min() get routed through dask.array.reductions.min() and will return plain arrays or float/int values.

When these operations exist as attributes, they can be called and will return unyt objects. i.e., :

import dask; import unyt
x_da = unyt_from_dask(dask.array.ones((10, 10), chunks=(2, 2)), unyt.m)
dask.array.min(x_da).compute()  #  returns a plain float
x_da.min().compute()  #  returns a unyt quantity

but when the functions do not exist as attributes, like dask.array.nanmin(), it is difficult to handle without manually wrapping all of those reductions functions and exposing them from within unyt (i.e., we would need to have a unyt.dask_array.nanmin() and similar for every operation).

To address this, #185 added a helper function, reduce_with_units. It is a simple helper function for calling a general dask.array reduction method on a unyt_dask_array to correctly handle units. It handles most cases, but it may be worth investigating other approaches.

@neutrinoceros neutrinoceros mentioned this issue Sep 9, 2022
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant