Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Termination of the Provider Node during a claim delete orphans resources #290

Closed
stevendborrelli opened this issue Oct 12, 2023 · 2 comments
Closed

Termination of the Provider Node during a claim delete orphans resources #290

stevendborrelli opened this issue Oct 12, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@stevendborrelli
Copy link

stevendborrelli commented Oct 12, 2023 •
edited
Loading

What happened?

If the node running an upjet provider is terminated/rebuilt while a claim is being deleted, the resources will be Orphaned in the cloud provider.

How can we reproduce it?

Overview

Test to see how Crossplane behaves during a Claim delete during a provider node termination. A user reported that resources are orphaned.

For testing:

  • create a 1 node EKS cluster borrelli-orphan, install UXP with EnvironmentConfigs enabled.
  • install platform-ref-aws on it and set up authentication
  • use EKS cluster borrelli-orphan to create another EKS cluster borrelli-orphan-target
  • Once borrelli-orphan-target is Ready, delete the claim
  • During claim deletion, terminate the single instance in the borrelli-orphan cluster.
  • Check if resources were correctly deleted

Environment

UXP 1.13.2

Platform-ref-aws master

kubectl get provider.pkg 
NAME                               INSTALLED   HEALTHY   PACKAGE                                                    AGE
crossplane-contrib-provider-helm   True        True      xpkg.upbound.io/crossplane-contrib/provider-helm:v0.15.0   23h
upbound-provider-aws-ec2           True        True      xpkg.upbound.io/upbound/provider-aws-ec2:v0.41.0           23h
upbound-provider-aws-eks           True        True      xpkg.upbound.io/upbound/provider-aws-eks:v0.41.0           23h
upbound-provider-aws-iam           True        True      xpkg.upbound.io/upbound/provider-aws-iam:v0.41.0           23h
upbound-provider-aws-rds           True        True      xpkg.upbound.io/upbound/provider-aws-rds:v0.41.0           23h
upbound-provider-family-aws        True        True      xpkg.upbound.io/upbound/provider-family-aws:v0.41.0        23h

Testing Setup

Create cluster borrelli-orphan with 1 large node:

apiVersion: aws.platformref.upbound.io/v1alpha1
kind: Cluster
metadata:
  name: borrelli-orphan
  namespace: default
  annotations:
    uptest.upbound.io/pre-delete-hook: testhooks/delete-release.sh
spec:
  id: borrelli-orphan
  parameters:
    version: "1.26"
    nodes:
      count: 1
      size: large
    services:
      operators:
        prometheus:
          version: "34.5.1"
        universal-crossplane:
          version: "1.12.2-up.1"
  writeConnectionSecretToRef:
    name: borrelli-orphan-kubeconfig

Once this cluster has been created, use it to deploy another cluster borrelli-orphan-target.

apiVersion: aws.platformref.upbound.io/v1alpha1
kind: Cluster
metadata:
  name: borrelli-orphan-target
  namespace: default
  annotations:
    uptest.upbound.io/pre-delete-hook: testhooks/delete-release.sh
spec:
  id: borrelli-orphan-target
  parameters:
    version: "1.26"
    nodes:
      count: 2
      size: large
    services:
      operators:
        prometheus:
          version: "34.5.1"
        universal-crossplane:
          version: "1.12.2-up.1"
  writeConnectionSecretToRef:
    name: borrelli-orphan-target-kubeconfig

borrelli-orphan-target managed resources:

NAME                                                            CHART                   VERSION       SYNCED   READY   STATE      REVISION   DESCRIPTION        AGE
release.helm.crossplane.io/borrelli-orphan-target-f4gwx-ccqgx   kube-prometheus-stack   34.5.1        True     True    deployed   1          Install complete   24m
release.helm.crossplane.io/borrelli-orphan-target-f4gwx-xlrp5   universal-crossplane    1.12.2-up.1   True     True    deployed   1          Install complete   5m57s

NAME                                                                      READY   SYNCED   EXTERNAL-NAME       AGE
securitygrouprule.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-7xc9g   True    True     sgrule-1460413336   24m
securitygrouprule.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-bxxfp   True    True     sgrule-491448244    24m

NAME                                                        READY   SYNCED   EXTERNAL-NAME           AGE
vpc.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-bqnvc   True    True     vpc-0936babf8b6a4496f   24m

NAME                                                                    READY   SYNCED   EXTERNAL-NAME           AGE
internetgateway.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-dkgw4   True    True     igw-05b7539e906f81a87   24m

NAME                                                          READY   SYNCED   EXTERNAL-NAME                       AGE
route.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-xk8lt   True    True     r-rtb-01afdb2cc4cbcccc31080289494   24m

NAME                                                                              READY   SYNCED   EXTERNAL-NAME                AGE
mainroutetableassociation.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-4zfdh   True    True     rtbassoc-033b63ca284f82246   24m

NAME                                                               READY   SYNCED   EXTERNAL-NAME           AGE
routetable.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-lmt6p   True    True     rtb-01afdb2cc4cbcccc3   24m

NAME                                                                          READY   SYNCED   EXTERNAL-NAME                AGE
routetableassociation.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-ldmtf   True    True     rtbassoc-098a90066f72fe24b   25m
routetableassociation.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-mqmlh   True    True     rtbassoc-0e9ecf9f2a6f58914   25m
routetableassociation.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-ps82k   True    True     rtbassoc-0e9c4685e13058ed8   25m
routetableassociation.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-qqdqn   True    True     rtbassoc-0a241dd90dcb2a802   25m

NAME                                                                  READY   SYNCED   EXTERNAL-NAME          AGE
securitygroup.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-9qtkk   True    True     sg-066ec39b434821c3c   25m

NAME                                                           READY   SYNCED   EXTERNAL-NAME              AGE
subnet.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-7757q   True    True     subnet-0d8eac6ef69cab8ba   25m
subnet.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-n4x4k   True    True     subnet-01665f5a9e0d17be6   25m
subnet.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-szstg   True    True     subnet-0c69ae4949b05a019   25m
subnet.ec2.aws.upbound.io/borrelli-orphan-target-f4gwx-wzcd8   True    True     subnet-09036d97797fdbbf1   25m

NAME                                                            READY   SYNCED   EXTERNAL-NAME                        AGE
cluster.eks.aws.upbound.io/borrelli-orphan-target-f4gwx-94f6q   True    True     borrelli-orphan-target-f4gwx-94f6q   21m

NAME                                                          READY   SYNCED   EXTERNAL-NAME                                           AGE
addon.eks.aws.upbound.io/borrelli-orphan-target-f4gwx-xn7pk   True    True     borrelli-orphan-target-f4gwx-94f6q:aws-ebs-csi-driver   21m

NAME                                                                READY   SYNCED   EXTERNAL-NAME                        AGE
clusterauth.eks.aws.upbound.io/borrelli-orphan-target-f4gwx-f9b77   True    True     borrelli-orphan-target-f4gwx-f9b77   21m

NAME                                                              READY   SYNCED   EXTERNAL-NAME                        AGE
nodegroup.eks.aws.upbound.io/borrelli-orphan-target-f4gwx-mhgb7   True    True     borrelli-orphan-target-f4gwx-mhgb7   21m

NAME                                                           READY   SYNCED   EXTERNAL-NAME                        AGE
policy.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-wxp9z   True    True     borrelli-orphan-target-f4gwx-wxp9z   25m

NAME                                                                         READY   SYNCED   EXTERNAL-NAME                                                   AGE
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-flj7q   True    True     borrelli-orphan-target-f4gwx-gj8w9-20231012122609671400000003   21m
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-jkfdx   True    True     borrelli-orphan-target-f4gwx-gj8w9-20231012122611993000000005   21m
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-nqvmd   True    True     borrelli-orphan-target-20231012123924668400000006               25m
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-nsns2   True    True     borrelli-orphan-target-f4gwx-gj8w9-20231012122611827400000004   21m
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-sfvds   True    True     borrelli-orphan-target-f4gwx-gj8w9-20231012122551655700000001   21m
rolepolicyattachment.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-smtpj   True    True     borrelli-orphan-target-f4gwx-v2p29-20231012122600156900000002   21m

NAME                                                         READY   SYNCED   EXTERNAL-NAME                        AGE
role.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-gj8w9   True    True     borrelli-orphan-target-f4gwx-gj8w9   21m
role.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-hjnqx   True    True     borrelli-orphan-target               7m35s
role.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-v2p29   True    True     borrelli-orphan-target-f4gwx-v2p29   21m

NAME                                                                          READY   SYNCED   EXTERNAL-NAME                                                                                                  AGE
openidconnectprovider.iam.aws.upbound.io/borrelli-orphan-target-f4gwx-m9dh6   True    True     arn:aws:iam::609897127049:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/CBA5B678A1FC5140897E0A9630876A9D   8m32s

borrelli-orphan instance and autoscaling group:

image

Delete the claim

$ kubectl get -f cluster-claim-target.yaml 
NAME                     SYNCED   READY   CONNECTION-SECRET                   AGE
borrelli-orphan-target   True     True    borrelli-orphan-target-kubeconfig   40m
$ kubectl delete -f cluster-claim-target.yaml 
cluster.aws.platformref.upbound.io "borrelli-orphan-target" deleted

Terminate the instance

image

Check claim and managed. Helm releases are left due to deletion ordering, which is unrelated.

kubectl get managed 
NAME                                                            CHART                   VERSION       SYNCED   READY   STATE      REVISION   DESCRIPTION        AGE
release.helm.crossplane.io/borrelli-orphan-target-f4gwx-ccqgx   kube-prometheus-stack   34.5.1        False    True    deployed   1          Install complete   46m
release.helm.crossplane.io/borrelli-orphan-target-f4gwx-xlrp5   universal-crossplane    1.12.2-up.1   False    False   deployed   1          Install complete   27m

Manually Validate Resources:

Resource external-name Deleted
VPC vpc-0936babf8b6a4496f No
IGW igw-05b7539e906f81a87 No
Subnet subnet-0d8eac6ef69cab8ba No
Nodegroup borrelli-orphan-target-f4gwx-mhgb7 No
Cluster borrelli-orphan-target-f4gwx-94f6q No
Role borrelli-orphan-target-f4gwx-gj8w9 No
Role borrelli-orphan-target-f4gwx-v2p29 No
OIDC borrelli-orphan-target No

Summary

During a node terminate/restart resources that are scheduled for deletion will be orphaned.
@stevendborrelli stevendborrelli added the bug Something isn't working label Oct 12, 2023
@turkenh
Copy link
Member

turkenh commented Oct 16, 2023

This looks related: crossplane-contrib/provider-upjet-aws#123

@jeanduplessis jeanduplessis added this to the 1.2 milestone Feb 3, 2024
@stevendborrelli
Copy link
Author

FYI, I tested this with the v0.47.x provider and it didn't replicate the behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stevendborrelli @jeanduplessis @turkenh