Created EKS Addon does not get saved to state if it does not become active

[flostadler]

Describe what happened

When creating an EKS Addon, the provider will first send the CreateAddon API call to AWS and then wait for the addon to become active.
Some addons, like corends, take a longer time to become active and might hit wait timeouts.
If the resource creation fails while waiting for the Addon to become active, the resource isn't saved to state.

Re-running pulumi up is now guaranteed to fail because Pulumi will wants to create the addon, even though it already exists in the cluster.

As a workaround, users either need to delete the addon from the cluster manually or import the Addon into Pulumi state.

Sample program

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Grab some values from the Pulumi configuration (or use default values)
const config = new pulumi.Config();
const vpcNetworkCidr = config.get("vpcNetworkCidr") || "10.0.0.0/16";

const env = "aws-addon-bug"

// Create a new VPC
const eksVpc = new awsx.ec2.Vpc("eks-vpc", {
    enableDnsHostnames: true,
    cidrBlock: vpcNetworkCidr,
});

const instanceRole = new aws.iam.Role('testrole', {
    assumeRolePolicy: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
            {
                Effect: 'Allow',
                Principal: {
                    Service: 'ec2.amazonaws.com',
                },
                Action: 'sts:AssumeRole',
            },
        ],
    }),
    managedPolicyArns: [
        'arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy',
        'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly',
        'arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy',
    ],
})

const eksCluster = new eks.Cluster(`${env}-cluster`, {
    vpcId: eksVpc.vpcId,
    authenticationMode: eks.AuthenticationMode.Api,
    corednsAddonOptions: {
        enabled: false,
    },
    createOidcProvider: true,
    enabledClusterLogTypes: ['api', 'audit', 'authenticator'],
    fargate: false,
    instanceRole: instanceRole,
    kubeProxyAddonOptions: {
        enabled: false,
    },
    nodeAssociatePublicIpAddress: false,
    privateSubnetIds: eksVpc.privateSubnetIds,
    publicSubnetIds: eksVpc.publicSubnetIds,

    skipDefaultSecurityGroups: false,
    skipDefaultNodeGroup: true,
    useDefaultVpcCni: false,
    version: '1.25',
});

const mng = new eks.ManagedNodeGroup(`${env}-managed-ng`, {
    cluster: eksCluster,
    instanceTypes: ['t3.medium'],
    scalingConfig: {
        desiredSize: 1,
        maxSize: 1,
        minSize: 1,
    },
    nodeRole: instanceRole,
});

const addonVersion = aws.eks.getAddonVersionOutput({
    addonName: 'coredns',
    kubernetesVersion: eksCluster.eksCluster.version,
    mostRecent: true,
}).version;

// takes ~ 15 minutes to create
new aws.eks.Addon(`${env}-cluster-coredns`, {
    clusterName: eksCluster.eksCluster.name,
    addonName: 'coredns',
    addonVersion: addonVersion,
    resolveConflictsOnCreate: 'OVERWRITE',
    resolveConflictsOnUpdate: 'OVERWRITE',
}, { customTimeouts: { create: '2m', update: '2m' }, dependsOn: [mng] });

// Export some values for use elsewhere
export const kubeconfig = eksCluster.kubeconfig;
export const vpcId = eksVpc.vpcId;

Log output

n/a

Affected Resource(s)

• aws.eks.Addon

Output of pulumi about

n/a

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[flostadler]

While working on pulumi/pulumi-eks#1509 I ran into this, but it occurred after updating pulumi-aws from v6.47.0 to v6.63.0: pulumi/pulumi-eks#1519 (comment)

This makes me believe that this is actually a regression. I'll do a bisect over the versions to confirm my suspicion

[flostadler]

I bisected the versions and it's indeed a regression that was introduced in v6.51.0.

That one includes multiple upstream upgrades and a bridge upgrade. Looking into these to find the root cause

[flostadler]

The upstream upgrades v5.64.0 and v5.65.0 do not include any suspect changes (no changes to any EKS resources). It doesn't repro in Terraform either.
So this is seems to be a regression on the pulumi side

[flostadler]

It's this: pulumi/pulumi-terraform-bridge#2696

We're not returning the partial state to the engine for init errors since enabling PRC. That's why Pulumi doesn't save the Addon to state when it's failing to initialize.

[pulumi-bot]

This issue has been addressed in PR #4898 and shipped in release v6.65.0.