Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue with stale connections to removed LB server #7194

Merged
merged 4 commits into from
Apr 4, 2023

Conversation

brandond
Copy link
Member

@brandond brandond commented Mar 31, 2023

Proposed Changes

  • Track LB connections through each server so that they can be closed when it is removed
  • Debounce kubernetes endpoint changes to that server connections do not thrash when new servers join the cluster
  • Update remotedialer to eliminate spurious errors when disconnecting from servers
  • Fix tests

Types of Changes

bugfix

Verification

  1. Run k3s with --debug
  2. Create multi-server cluster with at least one agent
  3. Delete a server from the cluster (using kubectl delete node)
  4. Note that agents log messages about closing LB connections to the deleted server

Testing

Linked Issues

User-Facing Change

The client load-balancer that maintains connections to active server nodes now closes connections to servers when they are removed from the cluster. This ensures that agent components immediately reconnect to a current cluster member.

Further Comments

@brandond brandond requested a review from a team as a code owner March 31, 2023 21:05
@brandond brandond force-pushed the close-lb-backend-removed branch 9 times, most recently from 55b2cc2 to 210f5b8 Compare April 1, 2023 00:53
cancel()
delete(disconnect, address)
logrus.Infof("Stopped tunnel to %s", address)
validEndpoint := map[string]bool{}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this all diffs very poorly, but it's just been indented inside the go func(){ goroutine

@brandond brandond requested a review from Oats87 April 1, 2023 01:29
go.mod Outdated Show resolved Hide resolved
@brandond brandond force-pushed the close-lb-backend-removed branch 3 times, most recently from 24fdc8b to 4e85024 Compare April 3, 2023 20:57
@brandond brandond requested review from dereknola and a team April 3, 2023 22:02
@brandond brandond force-pushed the close-lb-backend-removed branch 3 times, most recently from eb6289f to 9899eea Compare April 3, 2023 23:27
brandond added 3 commits April 3, 2023 23:44
Track LB connections through each server so that they can be closed when it is removed.

Signed-off-by: Brad Davidson <[email protected]>
@brandond brandond force-pushed the close-lb-backend-removed branch from 9899eea to 3da3db3 Compare April 3, 2023 23:44
@brandond brandond merged commit 2992477 into k3s-io:master Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants