Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal feature: Stateless scheduler #995

Open
Omrigan opened this issue Jun 26, 2024 · 0 comments
Open

Internal feature: Stateless scheduler #995

Omrigan opened this issue Jun 26, 2024 · 0 comments
Assignees

Comments

@Omrigan
Copy link
Contributor

Omrigan commented Jun 26, 2024

Problem description / Motivation

Currently, the scheduler acts as the source of truth for the cluster state. We'd like to change that because of the following reasons:

  1. Makes scheduler replication hard, as it is unclear how to reliably replicate that state.
  2. Every scheduler restart results in a state reset, thus a period of warm-up is needed to repopulate the state by the autoscaler-agent. This mechanism is responsible for additional complication of the components logic and protocol, namely:
    a. The existence of the Buffer values.
    b. The notion of lastPermit.
  3. Potentially, the upscaling communication diagram can be simplified to bypass the autoscaler-agent after the upscaling was approved by scheduler.

Feature idea(s) / DoD

  1. A scheduler can be replicated.
  2. A scheduler restart doesn't cause the sub-optimal scheduling decisions.
  3. We no longer have the concepts of Buffered resources and lastPermit.

Implementation ideas

The scheduler's state and autoscaler-agent ↔ scheduler protocol can be moved entirely to annotations on the associated VirtualMachine object.

Related work

The implementation of neondatabase/neon#8111 might allow us to significantly simplify autoscaler-agent (or even merge it's remaining functionality into other components).

Further information

There are more details in the internal RFC.

@sharnoff sharnoff self-assigned this Nov 18, 2024
sharnoff added a commit that referenced this issue Nov 18, 2024
I think the comments were just never updated when the autoscaler-agent
was moved from a per-pod sidecar to a per-node daemonset.

Found this while working on some background work for #995.
sharnoff added a commit that referenced this issue Nov 19, 2024
Similar to what was done in #1055, we need to explicitly add tolerations
to the scheduler to get it to be recreated more quickly on node failure.

This is particularly necessary because we don't have #995. We could wait
for that, but it's a lot of work, and this is a small thing we can do in
the meantime.

Fixes neondatabase/cloud#17298, part of neondatabase/cloud#14114.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants