Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't default RoleAssignmentName on machine templates #2111

Merged

Conversation

fiunchinho
Copy link
Contributor

/kind bug

What this PR does / why we need it:

While using SystemAssigned identity, I can't use more than 1 replica for my control plane. Or more than 1 replica on my MachineDeployments.

The problem is that AzureMachineTemplate has a RoleAssignmentName field for the name of the role assignment, which is then used when creating AzureMachines. But if more than one AzureMachine or VM is created out of this AzureMachineTemplate (i.e having more than 1 VM for the control plane, or even more than 1 replica in a MachineDeployment) the VM creation will fail with Azure API error RoleAssignmentUpdateNotPermitted saying Tenant ID, application ID, principal ID, and scope are not allowed to be updated.

Special notes for your reviewer:

I see that the templates used for e2e tests normally use only 1 replica for tests, that may explain why this hasn't been spotted so far.

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

Release note:

Fix 'SystemAssigned' identity by removing the defaulting of 'RoleAssignmentName' on 'AzureMachineTemplate' so that every 'AzureMachine'  defaults to a random 'RoleAssignmentName'.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 23, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @fiunchinho. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added area/provider/azure Issues or PRs related to azure provider sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 23, 2022
@fiunchinho fiunchinho force-pushed the roleassignment-machinetemplate branch from 76e0e2f to bcfbde4 Compare February 23, 2022 09:42
Copy link
Contributor

@CecileRobertMichon CecileRobertMichon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @shysank

@CecileRobertMichon
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 23, 2022
@shysank
Copy link
Contributor

shysank commented Feb 23, 2022

Ideally, we want to remove RoleAssignmentName from AzureMachineTemplate which needs a broader refactore, but for now, can we atleast add a validation to make sure no value gets set for that field?

@fiunchinho fiunchinho force-pushed the roleassignment-machinetemplate branch from 329b3ed to eb9f075 Compare February 23, 2022 21:06
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 23, 2022
@fiunchinho
Copy link
Contributor Author

I gave it a shot, please take a look to see if that's what you meant.

@shysank
Copy link
Contributor

shysank commented Feb 23, 2022

/lgtm
/assign @CecileRobertMichon
can you squash commits please?

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 23, 2022
@@ -92,5 +104,9 @@ func (r *AzureMachineTemplate) ValidateDelete() error {

// Default implements webhookutil.defaulter so a webhook will be registered for the type.
func (r *AzureMachineTemplate) Default() {
r.Spec.Template.Spec.SetDefaults()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than changing this here, shouldn't we update the func (s *AzureMachineSpec) SetDefaults() method directly in api/v1beta1/azuremachine_default.go? That SetDefaults() method is also invoked in api/v1beta1/azuremachine_webhook.go (in the func (m *AzureMachine) Default() method)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to default the RoleAssignmentName to a random UUID in the AzureMachine object, that's fine because all VMs would have a different one. It only doesn't make sense at the AzureMachineTemplate level.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation, makes sense

@jackfrancis
Copy link
Contributor

So the core issue here seems to be that a static value in a MachineTemplate will render as-is on each Machine resource using it. Surely there are other such examples of machine-unique configuration, how is that handled?

There seems to be something fundamentally broken in the type spec here. What is the ideal way to define machine-specific configuration as distinct from MachineTemplate configuration? Should such properties even be present in the MachineSpec type definition at all?

cc @CecileRobertMichon @shysank

@fiunchinho fiunchinho force-pushed the roleassignment-machinetemplate branch from eb9f075 to 51b2117 Compare February 24, 2022 09:00
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 24, 2022
@fiunchinho
Copy link
Contributor Author

fiunchinho commented Feb 24, 2022

Can you squash commits please?

done

Copy link
Contributor

@jackfrancis jackfrancis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 24, 2022
@CecileRobertMichon
Copy link
Contributor

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CecileRobertMichon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 24, 2022
@k8s-ci-robot k8s-ci-robot merged commit fb031f0 into kubernetes-sigs:main Feb 24, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.2 milestone Feb 24, 2022
@fiunchinho fiunchinho deleted the roleassignment-machinetemplate branch February 24, 2022 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/azure Issues or PRs related to azure provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants