Instance Profile doesn't attach roles or tags:

[empath-nirvana]

What happened?

I created an instance profile and no roles or tags were attached, even though it says it successfully synced

How can we reproduce it?

create role and instance profile with role and tags attached to instance profile and it will have neither a role nor tags attached.

apiVersion: iam.aws.crossplane.io/v1alpha1
kind: InstanceProfile
metadata:
  name: control-plane-node-profile-prod-use2
spec:
  deletionPolicy: Orphan
  forProvider:
    role: control-plane-node-role-prod-use2
    roleRef:
      name: control-plane-node-role-prod-use2
    tags:
    - key: Name
      value: control-plane-node-role-prod-use2
  providerConfigRef:
    name: control-plane

apiVersion: iam.aws.crossplane.io/v1beta1
kind: Role
metadata:
  name: control-plane-node-role-prod-use2
spec:
  deletionPolicy: Orphan
  forProvider:
    assumeRolePolicyDocument: '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
    description: Role for Nodes
    maxSessionDuration: 3600
    path: /
    tags:
    - key: Name
      value: control-plane-node-role-prod-use2
  providerConfigRef:
    name: control-plane

{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "control-plane-node-profile-prod-use2",
        "Arn": "arn:aws:iam::<account_number>:instance-profile/control-plane-node-profile-prod-use2",
        "CreateDate": "2022-09-16T17:24:09+00:00",
        "Roles": [],
        "Tags": []
    }
}

What environment did it happen in?

Crossplane version: crossplane/provider-aws-controller:v0.29.0
eks version 1.23

[haarchri]

is this error still valid ?
i can see in our environment everything is working as expected

Status:
  At Provider:
    Arn:                    arn:aws:iam::xxxx:instance-profile/karpenter
    Create Date:            2022-08-26T10:20:16Z
    Instance Profile ID:    AIPAV32PT5M2E5JSY77CU
    Instance Profile Name:  karpenter
    Roles:
      Arn:                          arn:aws:iam::xxxx:role/karpenter
      Assume Role Policy Document:  %7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Service%22%3A%22ec2.amazonaws.com%22%7D%2C%22Action%22%3A%22sts%3AAssumeRole%22%7D%5D%7D
      Create Date:                  2022-08-08T15:05:23Z
      Path:                         /
      Role ID:                      AROAV32PT5M2NRR3FOVOJ
      Role Name:                    karpenter
  Conditions:
    Last Transition Time:  2022-08-26T10:20:16Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-08-26T10:20:16Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
Events:                    <none>

[gmykhailiuta]

Can reproduce this bug with provider-aws v0.33.0 and crossplane 1.10.1 using next steps:

1. Create instance profile with role attached using crossplane.
2. Remove role from instance profile using awscli.
3. Crossplane does not fix nor report any problems with profile despite the fact that it has no roles attached any more.

Workaround so far is to recreate InstanceProfile resource.

[badams055]

Bumping this issue we are also seeing this behavior the only workaround is to put the resource in a composition and use: fromField: Required So that the instance profile is never created before the role is ready.

[bobh66]

I saw something similar when I started using this resource last week, and much of the odd behavior is due to the way the AWS API is implemented, combined with the fact that it's a v1alpha resource so it's not going to support updates or late initialization, and reconciliation will not work beyond "does the resource exist or not?". The create process towards AWS is required to be in two stages - first create the InstanceProfile resource and then attach the Role to it. If the Role is not available when the InstanceProfile gets created, it will never be attached and it's not considered to be an error condition.

There are some quirks related to the Role/RoleRef/RoleSelector. Role is a "required" field but it will get populated from the RoleRef or RoleSelector as long as one of the three is present. So if you specify a Role that doesn't exist yet, that passes the API checks because the Role has a value. The InstanceProfile resource gets created in AWS and the role association fails (silently!) so you end up with an InstanceProfile with no Role attached.

If you specify a roleRef or a roleSelector that won't resolve because the Role doesn't exist, the Role does not get populated and the reconciliation fails until the Resolver is successful. In that case it should "wait" for the Role to get created and then associate it properly.

Also if you specify a Role directly and the Role does not exist, you currently can't delete the InstanceProfile because the preDelete() function fails to disassociate the Role that doesn't exist and it fails the delete action. The only way around this is to either manually create the Role and associate it with the InstanceProfile to allow the preDelete() to succeed, or remove the finalizer from the InstanceProfile object and manually delete the InstanceProfile in AWS.

Clearly there is some room for improvement in the current implementation - the Role association should not succeed if the Role doesn't exist and the role disassociation should not fail the delete processing when the Role doesn't exist. I'll see if I can open a PR for those two issues.

[bobh66]

I ran a test and found that it does "fail" the create step when the specified Role cannot be attached to the InstanceProfile:

2023-03-22T01:03:06.285Z	DEBUG	provider-aws	Cannot create external resource	{"controller": "managed/instanceprofile.iam.aws.crossplane.io", "request": "/testip1badrole", "uid": "f66f64e6-67df-4202-b908-aa071589602f", "version": "120848886", "external-name": "", "error": "NoSuchEntity: The role with name mytestrole cannot be found.\n\tstatus code: 404, request id: cb603204-5b0a-40d1-a9d9-601f2bd5068f"}

but since the InstanceProfile does exist in the Observe step, it gets marked as Synced/Ready. It's just not completely Ready.

We could modify postObserve() to have it check for the presence of a Role in status.atProvider.roles[] but if it's not there what do we do? Report it as "resouce does not exist"? Then the Create() step will be reexecuted and fail with a 409 Conflict. Rerun the postCreate() step to hopefully attach the role?

This may not be completely solvable without upgrading the resource to v1beta to allow it to do updates.

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[harveyxia]

/fresh

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.