Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data_store_mgr: clear data from workflows that no longer have active subscriptions #585

Open
oliver-sanders opened this issue Apr 22, 2024 · 0 comments
Assignees
Milestone

Comments

@oliver-sanders
Copy link
Member

oliver-sanders commented Apr 22, 2024

We need the top-level workflow data from every workflow in a user's ~/cylc-run directory in the UIS data store, e.g:

  • Workflow ID
  • Host
  • Port
  • etc

Additionally, we require any other objects that the user has requested, e.g:

  • Tasks
  • Jobs
  • Graph nodes/edges
  • etc

When a user navigates away from a workflow, we no longer need to preserve these Tasks, Jobs, etc in the store and should purge it in due course to avoid stopped workflow data accumulating.

One approach would be to set a timer when the last subscription for a workflow is closed [1]. Unless another subscription on the workflow [1] is opened when this timer expires, then we purge the data.

This issue is similar to #568 which is about reducing the number of Protobuf "topics" we subscribe to as this issue also requires the monitoring of active subscriptions. This issue differs in that it is about clearing the data in the store that we have already acquired but is no longer of use rather than reducing the amount we request.

[1] Excluding top-level only subscriptions (see above) as this data is always required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants