Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds filters, compressors and serializer props to Array #2652

Merged
merged 7 commits into from
Jan 6, 2025

Conversation

normanrz
Copy link
Member

@normanrz normanrz commented Jan 5, 2025

Adds filters, compressors, serializer properties to the Array and AsyncArray classes. These properties mirror the kwargs that users provide in create_array, i.e. inner codecs in sharding are surfaced. Array.info is adapted accordingly. Fixes #2633 #2649.

I also fixed a bug where strings/bytes dtypes couldn't be used with sharding due to validation errors.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • GitHub Actions have all passed

@normanrz normanrz self-assigned this Jan 5, 2025
@normanrz normanrz added this to the After 3.0.0 milestone Jan 5, 2025

if self._filters is not None:
if len(self._filters) > 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this prevent arrays with no filters at all from displaying the fact that they have no filters via the .info method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likely out of scope, since my question is really about the pre-existing behavior of this function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "Filters : ... " line wasn't shown if filters was None, but it was shown if filters was (). Now, it is not shown if None or ().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we want to explicitly show the value of filters, even if it's ()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could. I don't have a strong preference here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it's better to show a static set of properties with this function, but if that's annoying to implement here we can defer for a separate PR

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about "Serializer", which doesn't exist for v2 arrays?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe None there? in principle there's nothing stopping someone from implementing a v2-compatible BytesCodec in numcodecs, in which case we might end up with v2 arrays with a serializer

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, there is no place in the v2 metadata for a serializer.
I opted for not showing the serializer field in the Array.info.

Copy link
Contributor

@d-v-b d-v-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

happy to see some simplified tests via these changes!

@normanrz normanrz merged commit 9138528 into main Jan 6, 2025
33 checks passed
@normanrz normanrz deleted the feat/filters-prop branch January 6, 2025 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add compressors, filters, serializer to Array as attributes
2 participants