Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunk transfer functions #463

Merged
merged 3 commits into from
Dec 12, 2023
Merged

Chunk transfer functions #463

merged 3 commits into from
Dec 12, 2023

Conversation

talonchandler
Copy link
Collaborator

@talonchandler talonchandler commented Dec 11, 2023

Fixes #460. Supersedes #461. Tested on @dsundarraman's reconstructions.

This PR chunks the transfer functions along YX, matching the reconstruction chunking.

Details: Writing to a .zarr with __setitem__ (i.e. dataset['group_name'] = myarray) uses a single chunk, so requesting a large reconstruction failed when the chunk exceeded 2147483647 bytes. The fix is to use create_image instead of __setitem__ so that we can set the chunk size.

Notes on tests: my first commit was a test on a large transfer function that failed locally, and my second commit fixed the problem and made the test pass locally. Unfortunately, the github action fails on this test, likely because it uses too much memory(?). My third commit reverts the test.

Copy link

codecov bot commented Dec 11, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d31ad91) 9.45% compared to head (1c6f81e) 9.45%.

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #463   +/-   ##
=====================================
  Coverage   9.45%   9.45%           
=====================================
  Files         29      29           
  Lines       4592    4592           
=====================================
  Hits         434     434           
  Misses      4158    4158           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@talonchandler talonchandler marked this pull request as ready for review December 11, 2023 22:04
@talonchandler talonchandler added this pull request to the merge queue Dec 12, 2023
Merged via the queue into main with commit b9ccc93 Dec 12, 2023
12 checks passed
@talonchandler talonchandler deleted the chunk-transfer-functions branch December 12, 2023 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Codec buffer limit error during reconstruction
3 participants