Secure bootstrapping for capz machines

[shysank]

What type of PR is this?
/kind feature

What this PR does / why we need it:

This PR provides an option to secure bootstrap data for capz machines. This is similar to the way capa works right now. On a high level, it works as follows: If SecureBootstrapEnabled is set to true,

1. Creates a azure key vault during cluster reconciliation
2. During machine reconciliation, saves the bootstrap data as secrets in the vault created in (1). The data is split into chunks to make sure the secret doesn't exceed azure keyvault limits.
3. Creates a cloudinit boothook script that: fetches the secrets created in (2) -> write them to a file -> restart cloudinit with the file created as source
4. As the vm is set to provision, set custom data to the script created in (3)

If SecureBootstrapEnabled is set to false (or not set), there is no change with the current behaviour.

Limitations:

1. This works only with UserAssigned identity. This is because the identity requires key vault administrator roles for reading and deleting secrets. System assigned identity is not possible since we can only assign role after vm provisioning.
2. Cluster identity also requires key vault administrator roles for creating vault and secrets.
3. SecureBootstrapEnabled works for the whole cluster, and not for individual machines (unlike capa). This is because we need to create a KeyVault for managing secrets, and doing it at the machine level will be too cumbersome. We could probably add an override at machine level to disable secure bootstrapping.

** Credits to @randomvariable and the CAPA team for the original implementation **

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #915 (partially)

Special notes for your reviewer:

I have split the pr into logical commits for easier review.

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

Secure bootstrapping for capz machines

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from shysank after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@shysank: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-apidiff	0e6debc	link	false	/test pull-cluster-api-provider-azure-apidiff

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[shysank]

/test pull-cluster-api-provider-azure-e2e-exp

[shysank]

cc @CecileRobertMichon

[CecileRobertMichon]

@shysank I haven't gone through the code yet but just from looking at the PR description, my initial questions are:

• Seems like this is making an assumption on cloud-init, what about other bootstrap mechanisms that don't leverage cloud-init, like cloudbase init for Windows nodes, Flatcar OS and ignition, or even other bootstrap providers outside of CABPK, like Talos?
• How does this overlap with Cleaner separation of kubeadm and machine bootstrapping cluster-api#5294 ?

Perhaps it would make sense to start with a design proposal since this is such a big change/feature.

[shysank]

• Seems like this is making an assumption on cloud-init, what about other bootstrap mechanisms that don't leverage cloud-init, like cloudbase init for Windows nodes, Flatcar OS and ignition, or even other bootstrap providers outside of CABPK, like Talos?
• How does this overlap with Cleaner separation of kubeadm and machine bootstrapping cluster-api#5294 ?

The goal of this pr is to provide a way to improve the security of bootstrapping mechanism that leverage cloud init only; bring capz on par with capa in terms of bootstrap data security albeit with some limitations (as mentioned in the pr description). This is not a generic solution to support other cloud initializers, perhaps we could implement UserdDataResolver for each cloud initializer, but we need more foundational changes to our design as mentioned in #5294 for a more robust solution . IMO, this is more of a stop gap solution until we are able separate kubeadm and machine bootstrapping.

Perhaps it would make sense to start with a design proposal since this is such a big change/feature.

I'll work on creating a design proposal.

[k8s-ci-robot]

@shysank: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[jackfrancis]

@CecileRobertMichon are you able to assess if we should try to salvage this PR?

[CecileRobertMichon]

I believe @sonasingh46 was looking into it https://kubernetes.slack.com/archives/CEX9HENG7/p1658756031736239

Regardless, we should start with a proposal as noted above and open a new PR once we're ready.

/close

[k8s-ci-robot]

@CecileRobertMichon: Closed this PR.

In response to this:

I believe @sonasingh46 was looking into it https://kubernetes.slack.com/archives/CEX9HENG7/p1658756031736239

Regardless, we should start with a proposal as noted above and open a new PR once we're ready.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sonasingh46]

I will work to raise a proposal PR for this.