Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Ensemble.batch use_map kwarg in favor of an implicit check for divisions #363

Open
dougbrn opened this issue Feb 1, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@dougbrn
Copy link
Collaborator

dougbrn commented Feb 1, 2024

use_map indicates whether to use a map_partitions call or just a general groupby when grouping lightcurves. The original motivation for this keyword was due to the fact that we didn't have a way to know if lightcurve cohesion would be true or not. I believe we're at the stage where if divisions are known, there's relatively high confidence that lightcurve cohesion is true (the only exception being if someone loads a non-lightcurve cohesive dataset into a dask dataframe from parquet, setting divisions using the calc_divisions flag, and then loading that dask dataframe into tape). I propose we do away with use_map and instead check if divisions are known, and in the case they are just automatically do a map_partitions call. The general groupby can be saved for the case where divisions are not known, as it's a more general/robust call.

@dougbrn dougbrn added the enhancement New feature or request label Feb 1, 2024
@dougbrn dougbrn self-assigned this Mar 11, 2024
@dougbrn dougbrn removed their assignment Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant