Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy: initialization wait for service and endpoint handlers synced #126532

Merged
merged 1 commit into from
Aug 14, 2024

Conversation

wedaly
Copy link
Member

@wedaly wedaly commented Aug 3, 2024

What type of PR is this?

/kind bug

What this PR does / why we need it:

Ensure kube-proxy waits for the services/endpointslices informer caches to be synced and all pre-sync events delivered before setting isInitialized=true. Otherwise, in clusters with many services, some services may be missing from svcPortMap when kube-proxy starts (e.g. during daemonset rollout). This can cause kube-proxy to temporarily remove service DNAT rules and then skip cleanup of UDP conntrack entries to a service VIP.

Which issue(s) this PR fixes:

Fixes #126468

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix race condition in kube-proxy initialization that could blackhole UDP traffic to service VIP.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

N/A

Ensure kube-proxy waits for the services/endpointslices informer
caches to be synced *and* all pre-sync events delivered before
setting isInitialized=true. Otherwise, in clusters with many services,
some services may be missing from svcPortMap when kube-proxy starts
(e.g. during daemonset rollout). This can cause kube-proxy to temporarily
remove service DNAT rules and then skip cleanup of UDP conntrack entries
to a service VIP.

Resolves: kubernetes#126468
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 3, 2024
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.31 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.31.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Sat Aug 3 16:50:49 UTC 2024.

@k8s-ci-robot k8s-ci-robot added area/kube-proxy cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/network Categorizes an issue or PR as relevant to SIG Network. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 3, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Aug 3, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @wedaly. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 3, 2024
@aojea
Copy link
Member

aojea commented Aug 5, 2024

/ok-to-test
/assign @aojea @danwinship

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 5, 2024
@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 5, 2024
@aojea
Copy link
Member

aojea commented Aug 5, 2024

@wedaly there are another two handlers for Nodes and ServiceCIDRs that may face similar problems, do you mind fixing all of them ?


EDIT Nevermind, we want to backport this, so better leave it this way

/lgmt

@danwinship for approval

@wedaly
Copy link
Member Author

wedaly commented Aug 5, 2024

EDIT Nevermind, we want to backport this, so better leave it this way

makes sense! I'm also happy to update the other handlers in a follow-on PR.

@aojea
Copy link
Member

aojea commented Aug 6, 2024

This is milestone v1.32 and we backport later, since is not a new bug or a regression

/milestone v1.32
/approve

This is a bug that was fixed in 1.27 already in most of the controllers #113985
, just nobody realized kube-proxy needed to be updated at that time , thanks @wedaly for fixing it

@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Aug 6, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, wedaly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2024
wedaly added a commit to wedaly/kubernetes that referenced this pull request Aug 6, 2024
Follow-on from kubernetes#126532
to wait for pre-sync events delivered for the remaining two informers
in kube-proxy (node and serviceCIDR).
@wedaly
Copy link
Member Author

wedaly commented Aug 6, 2024

EDIT Nevermind, we want to backport this, so better leave it this way

makes sense! I'm also happy to update the other handlers in a follow-on PR.

follow-on PR is here: #126561

@aojea
Copy link
Member

aojea commented Aug 14, 2024

/retest

@aojea
Copy link
Member

aojea commented Aug 14, 2024

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 14, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 4b65fb851918241dcbc4c07fd9088824b808e367

@k8s-ci-robot k8s-ci-robot merged commit 201bdaa into kubernetes:master Aug 14, 2024
14 checks passed
hungnguyen243 pushed a commit to hungnguyen243/kubernetes that referenced this pull request Aug 16, 2024
Follow-on from kubernetes#126532
to wait for pre-sync events delivered for the remaining two informers
in kube-proxy (node and serviceCIDR).
k8s-ci-robot added a commit that referenced this pull request Aug 28, 2024
…32-upstream-release-1.29

Automated cherry pick of #126532: kube-proxy: initialization wait for service and endpoint
k8s-ci-robot added a commit that referenced this pull request Aug 28, 2024
…32-upstream-release-1.28

Automated cherry pick of #126532: kube-proxy: initialization wait for service and endpoint
k8s-ci-robot added a commit that referenced this pull request Aug 28, 2024
…32-upstream-release-1.31

Automated cherry pick of #126532: kube-proxy: initialization wait for service and endpoint
k8s-ci-robot added a commit that referenced this pull request Aug 28, 2024
…32-upstream-release-1.30

Automated cherry pick of #126532: kube-proxy: initialization wait for service and endpoint
SoulPancake pushed a commit to SoulPancake/k8s that referenced this pull request Sep 11, 2024
Follow-on from kubernetes#126532
to wait for pre-sync events delivered for the remaining two informers
in kube-proxy (node and serviceCIDR).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kube-proxy cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kube-proxy: initialization check race leads to stale UDP conntrack
4 participants