Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Wait and requeue if LB not deleted #2122

Merged
merged 1 commit into from
Oct 16, 2024
Merged

🐛 Wait and requeue if LB not deleted #2122

merged 1 commit into from
Oct 16, 2024

Conversation

EmilienM
Copy link
Contributor

@EmilienM EmilienM commented Jun 11, 2024 •
edited
Loading

What this PR does / why we need it:

If the LB that is being deleted when a cluster is deleted, it'll go
through the PENDING_DELETE state and at this stage there is nothing we
can do but wait for the LB to be actually deleted.

If the LB is in that state during the deletion, let's just return no
error but request a reconcile after some time.

Which issue(s) this PR fixes:
Fixes #2124
Fixes #2121

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 11, 2024
@netlify Netlify
Copy link

netlify bot commented Jun 11, 2024 •
edited
Loading

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name Link
🔨 Latest commit 7ef114b
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-cluster-api-openstack/deploys/670fc9870a30d10008a5364e
😎 Deploy Preview https://deploy-preview-2122--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@EmilienM EmilienM changed the title Reconcile with wait after LB cleanup 🐛 Reconcile with wait after LB cleanup Jun 11, 2024
@EmilienM
Copy link
Contributor Author

/hold
I want to confirm we don't see the error in the logs.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2024
@EmilienM EmilienM self-assigned this Jun 11, 2024
@EmilienM
Copy link
Contributor Author

/hold cancel
I checked the logs and no more error of that type.

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/2122/pull-cluster-api-provider-openstack-e2e-test/1800607078713135104/artifacts/clusters/bootstrap/logs/capo-system/capo-controller-manager/capo-controller-manager-7d957cb6cd-5dgc2/manager.log

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 11, 2024
@jichenjc
Copy link
Contributor

/test pull-cluster-api-provider-openstack-e2e-test

@jichenjc
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jun 12, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 12, 2024
@EmilienM EmilienM changed the title 🐛 Reconcile with wait after LB cleanup Wait and requeue if LB is in PENDING_DELETE Jun 12, 2024
@EmilienM EmilienM mentioned this pull request Jun 12, 2024
Closed
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 12, 2024
@EmilienM EmilienM changed the title Wait and requeue if LB is in PENDING_DELETE Wait and requeue if LB + its ports not deleted Jun 12, 2024
@EmilienM EmilienM changed the title Wait and requeue if LB + its ports not deleted 🐛 Wait and requeue if LB + its ports not deleted Jun 12, 2024
@EmilienM EmilienM requested a review from mdbooth June 13, 2024 01:04
@EmilienM
Copy link
Contributor Author

@mdbooth ready for review when time permits. Maybe over-engineered but it does the job apparently. Feedback is open.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 1, 2024
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 26, 2024
@EmilienM
Copy link
Contributor Author

@mdbooth what to do with this one? It's been open for a long time.

@mdbooth
Copy link
Contributor

mdbooth commented Oct 16, 2024

What state did we leave it in? Is it about ready to go?

@mdbooth mdbooth added this to the v0.11 milestone Oct 16, 2024
mdbooth
mdbooth requested changes Oct 16, 2024
View reviewed changes
Copy link
Contributor

@mdbooth mdbooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the unnecessary check, but apart from that this is good to go imho.

pkg/cloud/services/loadbalancer/loadbalancer.go Show resolved Hide resolved
pkg/cloud/services/loadbalancer/loadbalancer.go Outdated Show resolved Hide resolved
pkg/cloud/services/loadbalancer/loadbalancer.go Outdated
@@ -658,32 +665,86 @@ func (s *Service) ReconcileLoadBalancerMember(openStackCluster *infrav1.OpenStac
return nil
}

func (s *Service) DeleteLoadBalancer(openStackCluster *infrav1.OpenStackCluster, clusterResourceName string) error {
func (s *Service) checkIfLbPortsExist(openStackCluster *infrav1.OpenStackCluster) (bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See below: we don't need this any more. In the context of the other fix (waiting until the LB is gone) this results in unnecessary API calls. It also depends on internal Octavia implementation details which aren't part of the API, so it adds additional future failure modes without any benefit. I'd very much prefer we removed this and the new function in port.go.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change in port.go was to address #2121 - I'm not sure that removing it will fix the issue with just the reconcile because it takes a few seconds for Octavia to remove ports when LB is being deleted but we'll see.

@EmilienM EmilienM changed the title 🐛 Wait and requeue if LB + its ports not deleted 🐛 Wait and requeue if LB not deleted Oct 16, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 16, 2024
@EmilienM
Copy link
Contributor Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2024
@EmilienM
Wait and requeue if LB is in PENDING_DELETE
7ef114b 
If the LB that is being deleted when a cluster is deleted, it'll go
through the PENDING_DELETE state and at this stage there is nothing we
can do but wait for the LB to be actually deleted.

If the LB is in that state during the deletion, let's just return no
error but request a reconcile after some time.
mdbooth
mdbooth reviewed Oct 16, 2024
View reviewed changes
Copy link
Contributor

@mdbooth mdbooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 16, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mdbooth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 28cccb9 into kubernetes-sigs:main Oct 16, 2024
9 checks passed
@mdbooth mdbooth deleted the lb branch October 16, 2024 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdbooth mdbooth mdbooth approved these changes

@jichenjc jichenjc Awaiting requested review from jichenjc

Assignees

@mdbooth mdbooth

@EmilienM EmilienM

@jichenjc jichenjc

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
CAPO Roadmap
Archived in project
Milestone
v0.11
Development

Successfully merging this pull request may close these issues.

Invalid state PENDING_DELETE of loadbalancer resource cluster delete with LB: network is removed too soon
4 participants
@EmilienM @jichenjc @mdbooth @k8s-ci-robot