Prevent relative expiry from emitting more events than can be processed #12002
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval.
rosstimothy
added
backport-required
scale
fspmarshall
approved these changes
Apr 20, 2022
smallinsky
reviewed
Apr 21, 2022
smallinsky
approved these changes
Apr 22, 2022
|
friendly ping @LKozlowski |
LKozlowski
approved these changes
Apr 25, 2022
rosstimothy
added a commit
that referenced
this pull request
Apr 26, 2022
…ed (#12002) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3) # Conflicts: # lib/cache/cache.go # lib/cache/cache_test.go
💚 All backports created successfully
Questions ?Please refer to the Backport tool documentation |
rosstimothy
added a commit
that referenced
this pull request
Apr 26, 2022
…ed (#12002) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3)
rosstimothy
added a commit
that referenced
this pull request
Apr 26, 2022
…ed (#12002) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3)
rosstimothy
added a commit
that referenced
this pull request
Apr 27, 2022
…ed (#12002) (#12245) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3)
rosstimothy
added a commit
that referenced
this pull request
May 2, 2022
…ed (#12002) (#12246) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3) # Conflicts: # lib/cache/cache.go # lib/cache/cache_test.go
rosstimothy
added a commit
that referenced
this pull request
May 2, 2022
…ed (#12002) (#12247) * Prevent relative expiry from emitting more events than can be processed This alters relative expiry in four ways: 1) Relative expiry is no longer exclusively run by the auth cache 2) Relative expiry no longer emits delete events 3) There is now a limit to the number of nodes removed per interval 4) Relative expiry now runs more frequently We can remove the need to emit any fake delete events during relative expiry by not running it exclusively in the auth cache. All caches will now run relative expiry themselves, even the components that don't watch for nodes - this is effectively a noop for them. To prevent the individual caches from getting too far out of sync, the expiry interval is set to a much smaller value and we limit the number of nodes being deleted per interval. (cherry picked from commit b8394b3)
Closed
rosstimothy
added a commit
that referenced
this pull request
Feb 15, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 15, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
github-actions bot
pushed a commit
that referenced
this pull request
Feb 15, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
github-actions bot
pushed a commit
that referenced
this pull request
Feb 15, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 17, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 17, 2024
When relative node expiry was originally implemented it only ran on Auth and propagated delete events downstream. However, that was quickly changed in #12002 because the event system could not keep up in clusters with high churn and would result in buffer overflow errors in downstream caches. The event system has been overhauled to use FanoutV2, which no longer suffers from the burst of events causing problems. The change to not propagate delete events also breaks any node watchers on the local cache from ever expiring the server since they never receive a delete event. This reverts some of the changes from #12002 such that relative expiry now only runs on the auth cache and emits delete events. Each relative expiry interval is however still limited to only remove up to a fixed number of nodes. Closes #37527
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This alters relative expiry in four ways:
We can remove the need to emit any fake delete events during relative
expiry by not running it exclusively in the auth cache. All caches
will now run relative expiry themselves, even the components that
don't watch for nodes - this is effectively a noop for them. To
prevent the individual caches from getting too far out of sync, the
expiry interval is set to a much smaller value and we limit the
number of nodes being deleted per interval.