How to create CustomResources in a loop?

[zentavr]

Describe the bug

I have a code like this:

import { Construct } from 'constructs';

export class EksClusterDeployment extends Construct {
    eksCluster: eks.Cluster;
    const nodeGroupTags: { [key: string]: any } = {};

    constructor(scope: Construct, id: string, props: EksClusterDeploymentProps) {
        this.eksCluster = new eks.Cluster(.......);

        // In the real life this array gets calculated, depends which subnets the operator had defined.
        availabilityZones = ['us-east-1a', 'us-east-1b', 'us-east-1c', 'us-east-1d', 'us-east-1f'];

        availabilityZones.forEach(function (az){
            let ng = this.eksCluster.addNodegroupCapacity(id + az.slice(-1), modOptions);

            // Compose the tags
            nodeGroupTags["k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/region"] = cdk.Stack.of(cls).region;
            nodeGroupTags["k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/zone"] = az;
            nodeGroupTags["k8s.io/cluster-autoscaler/node-template/label/topology.ebs.csi.aws.com/zone"] = az;
            nodeGroupTags["k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/region"] = cdk.Stack.of(cls).region;
            nodeGroupTags["k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/zone"] = az;

            // If I put console.log(nodeGroupTags) here - the output would be unique per iteration and this is I can see in my local console

            // Create custom resource which updates ASG tags
            new NodegroupASGModifier(cls, id + az.slice(-1), {
                cluster: cls.eksCluster,
                nodegroup: ng,
                tags: nodeGroupTags
            });
            // Creation and modification of MNG+ASG is done
        });
    }
}

export class NodegroupASGModifier extends Construct {
    constructor(scope: Construct, id: string, props: NodegroupASGModifierProps) {
        super(scope, id);

        const onEventHandler = new lambda.Function(this, id + '-onEventHandler', {
            timeout: cdk.Duration.seconds(60),
            handler: 'index.on_event',
            runtime: lambda.Runtime.PYTHON_3_10,
            code: lambda.Code.fromAsset(path.join(__dirname, '../..', 'resources/lambdas/elk-adds-extratags-to-asg'), {
                //bundling: {
                //    image: lambda.Runtime.PYTHON_3_10.bundlingImage
                //}
            })
        });
        const nodegroupName = (props.nodegroup.node.defaultChild as eks.CfnNodegroup).getAtt('NodegroupName').toString();

        // Allow to describe the Nodegroup and modify Autoscaling group
        onEventHandler.addToRolePolicy(new iam.PolicyStatement({
            actions: ['eks:DescribeNodegroup'],
            resources: [
                cdk.Stack.of(this).formatArn({
                    resource: 'nodegroup',
                    service: 'eks',
                    resourceName: `${props.cluster.clusterName}/${nodegroupName}/*`
                })
            ]
        }));
        onEventHandler.addToRolePolicy(new iam.PolicyStatement({
            actions: [
                'autoscaling:DeleteTags',
                'autoscaling:CreateOrUpdateTags',
                'autoscaling:DescribeTags',
                'autoscaling:UpdateAutoScalingGroup',
            ],
            resources: ['*']
        }));

        // Creating CustomResource Provider
        const provider = new cr.Provider(this, id + '-ASGModProvider', {
            onEventHandler: onEventHandler,
            logRetention: logs.RetentionDays.TWO_WEEKS
        });

        console.log('===================================')
        console.log('onEventHandler: ' + id + '-onEventHandler')
        console.log('provider: ' + id + '-ASGModProvider');
        console.log('NodegroupASGModifierRes: ' + id + '--NodegroupASGModifier');
        console.log('Fixing NodeGroup: ' + nodegroupName.toString());
        console.log('Nodegroup: ' + props.nodegroup.nodegroupName)
        console.log(props.tags)
        console.log('===================================')
        // Invoke Custom Resource
        const NodegroupASGModifierRes = new cdk.CustomResource(this, id + '-NodegroupASGModifier', {
            serviceToken: provider.serviceToken,
            properties: {
                clusterName: props.cluster.clusterName,
                nodegroupName,
                tags: props.tags
            }
        });
    }
}

The problem is that when I check the generated template in cdk.out of in AWS Console, I can see that all the Custom Resources got the same parameters (tags in my case), while console.log() which I'd put across the code prints the unique values.

Expected Behavior

I would like to create unique custom resources in obj.forEach()

Current Behavior

When I have something inside obj.forEach() which needs to be created by CDK, it's not created properly.

Reproduction Steps

The code sample was provided in the block above

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0 (build 9fca790)

Framework Version

2.87.0

Node.js Version

v18.16.0

OS

MacOS

Language

Typescript

Language Version

No response

Other information

No response

[zentavr]

Do I need to use something from CloudFormation intrinsic functions?

The idea of adding the tags to autoscaling groups was found here. (Authored by @pahud)

[pahud]

Thank you for bringing up my previous sample. I appreciate that.

Looks like you are trying to create individual NG for the 5 AZs(['us-east-1a', 'us-east-1b', 'us-east-1c', 'us-east-1d', 'us-east-1f']), each NG has multiple custom tags applied. In this case I would prefer a single custom resource that takes care all tags for all NGs. That might be cleaner and easier to maintain.

But I am wondering what makes you to create NG for each az? I assume a NG should be cross-az?

As this issue is not a bug or feature request but general discussion, I am moving this to discussion but I would be happy to continue discuss your use case with you.

Also add @robertd in the loop.

[zentavr]

Hi @pahud. The reason why I am trying to implement managed nodegroup per AZ is the problem with the EBS driver + autoscaler the cluster also has. It was discussed for example here. The cluster has bitnami/rabbitmq helm chart and the PVs get created in the different zones, but the autoscaler cannot spin up the nodes for that when the nodegroup has all the subnets in the settings and is multi-AZ nodegroup.

Each autoscaling group which appears after Nodegroup gets created by CDK should have different tags, for example:
NodeGroup (onDemand) for zoneA:

{
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/region": "us-east-1"
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/zone": "us-east-1a",
     "k8s.io/cluster-autoscaler/node-template/label/topology.ebs.csi.aws.com/zone": "us-east-1a",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/region": "us-east-1",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/zone"]: "us-east-1a",
     "k8s.io/cluster-autoscaler/node-template/label/k8s.our-example.org/instance-type"]: "ON_DEMAND",
}

NodeGroup (onDemand) for zoneB:

{
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/region": "us-east-1"
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/zone": "us-east-1b",
     "k8s.io/cluster-autoscaler/node-template/label/topology.ebs.csi.aws.com/zone": "us-east-1b",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/region": "us-east-1",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/zone"]: "us-east-1b",
     "k8s.io/cluster-autoscaler/node-template/label/k8s.our-example.org/instance-type"]: "ON_DEMAND",
}

Also we have some ETLs which may run on spot instances and the tags are different for those ASGs. e.g.:

{
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/region": "us-east-1"
     "k8s.io/cluster-autoscaler/node-template/label/failure-domain.beta.kubernetes.io/zone": "us-east-1d",
     "k8s.io/cluster-autoscaler/node-template/label/topology.ebs.csi.aws.com/zone": "us-east-1d",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/region": "us-east-1",
     "k8s.io/cluster-autoscaler/node-template/label/topology.kubernetes.io/zone"]: "us-east-1d",
     "k8s.io/cluster-autoscaler/node-template/label/k8s.our-example.org/instance-type"]: "SPOT",
}

Like I said earlier, I'd used your example which modified maxInstanceLifetime, but I'm modifying the tags. The Role Policy code for the Lambda functions which does the modifications was copy-pasted from your example (it uses the certain resource) - that's why we have several Lambdas (code is the same, permissions are little bit different, parameters are little bit different).

[zentavr]

Seems like I'd found the issue. The problem was that nodeGroupTags was defined as const nodeGroupTags: { [key: string]: any } = {}; at the beginning of the class definition. I'd removed that and did let nodeGroupTags: { [key: string]: any } = {}; inside the .forEach() function.