Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming module: More fine-grained recomputes #92

Open
SimonHeybrock opened this issue Sep 2, 2024 · 0 comments
Open

Streaming module: More fine-grained recomputes #92

SimonHeybrock opened this issue Sep 2, 2024 · 0 comments

Comments

@SimonHeybrock
Copy link
Member

This is a note to get it off my mind, currently not a requirement that needs to be worked on!

ess.reduce.streaming.StreamProcessor currently takes an all-or-nothing approach to updating and re-computing: It is assumed that all "dynamic inputs" are updated at the same frequency. Then, everything (non-static) is recomputed, accumulated, and final results computed.

In practice, there may be dynamic inputs that change with different frequency (for examples fast detector event data updates alongside slow detector bank move), and we may thus want to reflect this in the way the workflow updates/recomputes. Pre-computation of static inputs may be replaced by a more dynamic mechanism that can handle nodes that updated infrequently (or not at all).

As this is potentially more risky, since it seems to be constituting a potentially more error-prone caching mechanism, I think we should not perform such a change until a clear need is identified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant