Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split load of detector/monitor from event-data load #153

Merged
merged 15 commits into from
Aug 7, 2024

Conversation

SimonHeybrock
Copy link
Member

@SimonHeybrock SimonHeybrock commented Jul 24, 2024

This addition is related to the new approach discussed in scipp/beamlime#215. It depends on a release of scipp/essreduce#53, i.e., CI will fail for now.

Edit: converted to draft, since I found one more dependency on event data that should not be there (for solid angle). -> This is fixed by the follow-up #155 (now merged into this branch). -> Fixes #154.
Also closes #39, closes #62.

By loading NXevent_data separately from the containing NXdetector or NXmonitor we increase flexibility, reduce overhead, and continue paving the path towards chunked processing of large files.

Before:

Image

After:

image

Notes:

  • By getting the detector IDs from the RawData[SampleRun], which is now the "empty" detector with no events, masks can be loaded and processed without loading potentially huge amounts of events. Even if the events will be loaded anyway this will improve parallelism, and will avoid reprocessing of masks if the event data changes.
  • Chunking loading of the event data (using Sciline's map-reduce) with the "after" graph will only load new events, which is also what we need to do for live-reduction. Other detector information as well as masks can be reused. In short, the graph is "wider", which is better.

@SimonHeybrock
Copy link
Member Author

I believe this was reviewed as part of #155.

@SimonHeybrock SimonHeybrock merged commit 6e5fa3a into main Aug 7, 2024
3 checks passed
@SimonHeybrock SimonHeybrock deleted the split-event-load branch August 7, 2024 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
1 participant