Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Push changes to ModuleContext down to the runner #1699

Closed
alecthomas opened this issue Jun 7, 2024 · 0 comments · Fixed by #1865
Closed

Push changes to ModuleContext down to the runner #1699

alecthomas opened this issue Jun 7, 2024 · 0 comments · Fixed by #1865
Assignees

Comments

@alecthomas
Copy link
Collaborator

alecthomas commented Jun 7, 2024

Currently we pull the ModuleContext once, but we really want some aspects of it to be dynamic, in particular secrets/config.

We'd need to change the Controller endpoint to be a streaming response, and use PG's pubsub support to notify the controllers of changes.

  1. Turn GetModuleContext() into a streaming RPC - this will also need to retry the streaming RPC in case the stream dies - potentially just use RetryStreamingClientStream if it's suitable.
  2. Block until the first ModuleContext arrives so the module has something to use.
  3. Inside controller.Service.GetModuleContext, periodically reconstruct the ModuleContext, hash it, and check against the previous version. If it's changed, send it. ModuleContext needs to be deterministically hashable.
  4. Add a new type to the modulecontext package that takes the gRPC client and refreshes an atomically (alecthomas/atomic.Value[T]) retrievable ModuleContext.
  5. Instead of adding the ModuleContext directly to the context, we'll add this self-refreshing type.
  6. Inside the go-runtime replace all direct uses of ModuleContext with this new type.
@github-actions github-actions bot added the triage Issue needs triaging label Jun 7, 2024
@ftl-robot ftl-robot mentioned this issue Jun 7, 2024
@alecthomas alecthomas added next Work that will be be picked up next and removed triage Issue needs triaging labels Jun 10, 2024
@github-actions github-actions bot removed the next Work that will be be picked up next label Jun 11, 2024
jonathanj-square added a commit that referenced this issue Jun 24, 2024
`GetModuleContext` will stream updated `ModuleContext` to the runner.
jonathanj-square added a commit that referenced this issue Jun 24, 2024
`GetModuleContext` will stream updated `ModuleContext` to the runner.
jonathanj-square added a commit that referenced this issue Jun 25, 2024
`GetModuleContext` will stream updated `ModuleContext` to the runner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants