Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix launch template reconciliation if bootstrap data secret cannot be read #4589

Conversation

AndiDog
Copy link
Contributor

@AndiDog AndiDog commented Oct 25, 2023

What type of PR is this?

/kind bug

What this PR does / why we need it:

When changing the machine pool's bootstrap config, e.g. KubeadmConfig/{a => b}, which is supported after CAPI's fix kubernetes-sigs/cluster-api#8667, CAPA had a bug where it would continue even after the new bootstrap secret cannot be read, for instance because it doesn't exist yet as the bootstrap provider still needs a few seconds to make it ready. That could lead to a catastrophic misconfiguration where the launch template's user data is empty, typically leading to new instances not getting bootstrapped at all. Detailed problem description in this comment.

This provides the obvious fix. I couldn't find a great place to test this, since ReconcileLaunchTemplate is mocked away in tests.

I tested this successfully by changing the bootstrap config reference and observing how CAPI sets the new data secret name, CAPA intermittently failing to read the new secret until it's ready, and only then CAPA writing a new launch template version with the new bootstrap secret's data.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Part of kubernetes-sigs/cluster-api#8858, but not a fix. Even with this patch, changing the bootstrap config object doesn't lead to an instance refresh (rolling out new nodes) because CAPA doesn't consider it a change. We need to discuss further how to solve the main issue.

Checklist:

  • squashed commits
  • includes documentation
  • adds unit tests
  • adds or updates e2e tests

Release note:

Fix launch template reconciliation if bootstrap data secret cannot be read

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 25, 2023
@k8s-ci-robot k8s-ci-robot requested review from faiq and vincepri October 25, 2023 13:02
@richardcase
Copy link
Member

/test ?

@k8s-ci-robot
Copy link
Contributor

@richardcase: The following commands are available to trigger required jobs:

  • /test pull-cluster-api-provider-aws-build
  • /test pull-cluster-api-provider-aws-test
  • /test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

  • /test pull-cluster-api-provider-aws-apidiff-main
  • /test pull-cluster-api-provider-aws-e2e
  • /test pull-cluster-api-provider-aws-e2e-blocking
  • /test pull-cluster-api-provider-aws-e2e-clusterclass
  • /test pull-cluster-api-provider-aws-e2e-conformance
  • /test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
  • /test pull-cluster-api-provider-aws-e2e-eks
  • /test pull-cluster-api-provider-aws-e2e-eks-gc
  • /test pull-cluster-api-provider-aws-e2e-eks-testing

Use /test all to run the following jobs that were automatically triggered:

  • pull-cluster-api-provider-aws-apidiff-main
  • pull-cluster-api-provider-aws-build
  • pull-cluster-api-provider-aws-test
  • pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@richardcase
Copy link
Member

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@AndiDog
Copy link
Contributor Author

AndiDog commented Oct 30, 2023

/test pull-cluster-api-provider-aws-e2e-eks

Copy link
Contributor

@cnmcavoy cnmcavoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2023
@AndiDog
Copy link
Contributor Author

AndiDog commented Oct 31, 2023

@richardcase The test pull-cluster-api-provider-aws-e2e-eks was waiting for an EKS cluster to be created and the result here in GitHub says "Pod got deleted unexpectedly". Do you have an idea for this issue? Can it be because of #4575 not merged into my PR?

@faiq
Copy link
Contributor

faiq commented Nov 1, 2023

/lgtm

@k8s-ci-robot k8s-ci-robot added do-not-merge/contains-merge-commits and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Nov 2, 2023
@AndiDog
Copy link
Contributor Author

AndiDog commented Nov 2, 2023

Trying the test again after having merged main (incl. possible test fix #4575)

/test pull-cluster-api-provider-aws-e2e-eks

@AndiDog AndiDog force-pushed the machine-pool-bootstrap-config-ref-rotation branch from cbeaab4 to 2b13c74 Compare November 2, 2023 12:21
@AndiDog
Copy link
Contributor Author

AndiDog commented Nov 2, 2023

@cnmcavoy @faiq I merged main and the E2E test worked this time. Switched to rebase since merge isn't allowed. Need another LGTM – no changes happened to my diff.

@faiq
Copy link
Contributor

faiq commented Nov 2, 2023

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 2, 2023
@faiq
Copy link
Contributor

faiq commented Nov 2, 2023

/test pull-cluster-api-provider-aws-e2e-eks

@faiq
Copy link
Contributor

faiq commented Nov 3, 2023

/approve

@Ankitasw
Copy link
Member

Ankitasw commented Nov 4, 2023

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ankitasw, faiq

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 4, 2023
@k8s-ci-robot k8s-ci-robot merged commit 07e91d2 into kubernetes-sigs:main Nov 4, 2023
7 checks passed
@AndiDog AndiDog deleted the machine-pool-bootstrap-config-ref-rotation branch November 6, 2023 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants