[draft] Accept Cubed arrays instead of dask #249
Conversation
| from xarray.core.parallelcompat import get_chunked_array_type | ||
|
|
||
| chunkmanager = get_chunked_array_type(array) |
There was a problem hiding this comment.
Obviously this approach would introduce a dependency on xarray, which presumably is not desirable.
There was a problem hiding this comment.
I'm fine just having dask_kwargs and cubed_kwargs instead of all this complexity.
There was a problem hiding this comment.
I probably should have just done that in xarray itself 😅
| # meta=array._meta, | ||
| align_arrays=False, | ||
| name=f"{name}-chunk-{token}", | ||
| # name=f"{name}-chunk-{token}", |
There was a problem hiding this comment.
_meta and name are dask-specific. Are they used for anything important here or just for labelling tasks in the graph?
There was a problem hiding this comment.
if you don't provide meta, dask will try to figure it out and then break?
| else: | ||
| combine = partial(_grouped_combine, engine=engine, sort=sort) | ||
| combine_name = "grouped-combine" |
There was a problem hiding this comment.
There's a test for these names that will need to be fixed.
| @@ -1889,7 +1896,7 @@ def groupby_reduce( | |||
| axis_ = np.core.numeric.normalize_axis_tuple(axis, array.ndim) # type: ignore | |||
| nax = len(axis_) | |||
|
|
|||
| has_dask = is_duck_dask_array(array) or is_duck_dask_array(by_) | |||
| has_dask = is_chunked_array(array) or is_duck_dask_array(by_) | |||
There was a problem hiding this comment.
| has_dask = is_chunked_array(array) or is_duck_dask_array(by_) | |
| is_chunked = is_chunked_array(array) or is_chunked_array(by_) |
…axis to chunk identity fn
for more information, see https://pre-commit.ci
Very very rough changes to see what happens if you try to give flox a
cubed.Array. Not many changes are required to get the cubed array inputs all the way to the reduction step, but I have not yet been able to run it, so there might be additional incompatibilities that turn up.