feat(eks): pass helm chart values to aws-load-balancer-controller #29723

mtrspringer · 2024-04-04T11:47:00Z

Issue # (if applicable)

Reason for this change

Need to be able to disable aws-load-balancer-controller wafv2 behavior so it doesn't remove waf associations created by AWS FMS.

Description of changes

I added a values property to the AlbControllerOptions interface for passing optional values to the underlying helm chart.

Description of how you validated changes

I added two tests to validate my changes:

a test to check that values passed appear in the template output
a test to check that values passed do not override values currently set by the AlbController construct

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

lpizzinidev

Looks good, thanks! 👍
Left some minor suggestions for updating the documentation.

packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

packages/aws-cdk-lib/aws-eks/README.md

paulhcsun · 2024-04-19T21:29:39Z

packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

+   *
+   * @default - No values are provided to the chart.
+   */
+  readonly values?: {[key: string]: any};


Suggested change

readonly values?: {[key: string]: any};

readonly helmChartValues?: {[key: string]: any};

Can you rename this to helmChartValues? Just having this be values in this context is not clear what these values are being used for. Ideally I'd have like the property within Helm Chart to be named something other than values as well but unfortunately it's too late for that now.

yes that's fine with me. I was using the HelmChartProps as an guide but I can see the value in being specific within the scope of a broader construct that's not limited to helm already.

paulhcsun · 2024-04-19T21:32:35Z

packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

+   * values: {
+   *  autoscaling: false,
+   *  ingressClassParams: { create: true }
+   * }
+   *
+   * Note that the following values are set by the controller and cannot be overridden:
+   *  - clusterName
+   *  - serviceAccount.create
+   *  - serviceAccount.name
+   *  - region
+   *  - vpcId
+   *  - image.repository
+   *  - image.tag


Do we know the full set of values that can be passed in to the helmChart.values property? If so could we add some stronger typing checks to them? If not then could we add some validation for the values that cannot be overridden?

From your unit test case it seems like Helm Chart will just ignore these values but I would rather prevent these values from being set at all instead of allowing it to be passed in and have it silently fail.

i think the set of values varies (or can potentially vary) depending on the helm chart version. I think defining a type (e.g. AlbControllerHelmChartValues) will create a loose link to the helm chart that has the potential for breaking. Validating the passed values against the keys we know we explicitly set seems like a better approach to me.

Gotcha, thanks for clarifying. Then let's go with the second option to validate the passed values that cannot be overridden.

@paulhcsun i've updated with the requested changes. Assuming the suite passes, would you (or someone else) be able to run/update the integration tests? I ran into some issues with the teardown last time.

Hey @mtrspringer, thanks for making the changes! and ya for sure, I'll give that a run later today.

Hey @mtrspringer, apologies some other items came up. I will run the tests now.

It took a while to run on Friday and looks like it failed due to a credentials related error but it still seems to have updated the snapshots. I've pushed them so hopefully the codebuild will be happy with the new snapshots.

@paulhcsun thanks very much! yes it takes ages, looks like the build is passing though

Pull request has been modified.

paulhcsun · 2024-05-28T20:18:51Z

Hi @mtrspringer, apologies for another delay to getting around to reviewing this. I was oncall last week. Thank you for making the requested changes for validating the values that can be passed to helmChart.values.

Would you mind explaining what the changes within @aws-cdk/sdk-v2-to-v3-adapter are for? I don't remember seeing them during the initial review and I'm not quite sure how they're related to this change.

mtrspringer · 2024-06-07T15:53:16Z

@paulhcsun hi sorry i've been on vacation the last 2 weeks. I dont remember making any changes to those files, maybe they got included when i updated my branch via the github button? im happy to leave them out

mtrspringer · 2024-06-11T10:56:38Z

@paulhcsun looks like they were .d.ts and .js files that somehow got included. i've just removed them so this PR should be ready for review now

mtrspringer · 2024-07-09T10:34:32Z

hi @Necrokefalos im still waiting on a response as to what changes, if any, need to be made. good to know that the values also need to be modified for cert-manager as we are looking to adopt that sometime soon. in the meantime i think the integration tests need to be run again before the checks pass. @paulhcsun would you be able to do that again at your earliest convenience? thanks.

mtrspringer · 2024-07-17T14:19:01Z

@Necrokefalos not sure if this will work for your use-case (and it's hacky) but in the interim while this is being deliberated I realized we can "disable" alb controller behavior via the serviceAccount policy, which we can access via the construct node.findChild api:

    // TODO: remove once cdk supports passing values to albController Helm Chart:
    const {enableWafv2 = false} = albControllerValues;
    if (!enableWafv2) {
      const albControllerServiceAccount =
        this.eksCluster.albController!.node.findChild(
          'alb-sa'
        ) as eks.ServiceAccount;
      albControllerServiceAccount.addToPrincipalPolicy(
        new iam.PolicyStatement({
          effect: iam.Effect.DENY,
          actions: [
            'waf-regional:GetWebACL',
            'waf-regional:GetWebACLForResource',
            'waf-regional:AssociateWebACL',
            'waf-regional:DisassociateWebACL',
            'wafv2:GetWebACL',
            'wafv2:GetWebACLForResource',
            'wafv2:AssociateWebACL',
            'wafv2:DisassociateWebACL',
          ],
          resources: ['*'],
        })
      );
    }

i still have to test how this affects the controller's operation (i.e. can it gracefully handle the permission errors) but it might be easier than replacing the cdk.eks.Cluster.albController with a custom HelmChart implementation.

paulhcsun · 2024-08-01T20:30:17Z

Hi @mtrspringer, apologies for the late response I was away recently and just got back to this. I've started running the integ tests so hopefully i'll get those pushed today or tmr. I think when I last discussed with Kendra there was one design change for how we allow the users to update the helm chart values. I will confirm with her and get back asap. Apologies for the churn.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

paulhcsun · 2024-08-02T16:05:25Z

I've pushed the snapshot changes for the integ.alb-controller.js test however the test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js test currently fails with this error:

Tests:    1 failed, 921 total
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js in us-east-1
/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/shared/base-load-balancer.js:93
                throw new Error(`'denyAllIgwTraffic' may only be set on load balancers with ${enums_1().IpAddressType.DUAL_STACK} addressing.`);
                ^

Error: 'denyAllIgwTraffic' may only be set on load balancers with dualstack addressing.
    at new BaseLoadBalancer (/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/shared/base-load-balancer.js:93:23)
    at new NetworkLoadBalancer (/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/nlb/network-load-balancer.js:165:9)
    at Object.<anonymous> (/Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js:13:1)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:142:12)
    at node:internal/main/run_main_module:28:49

Node.js v21.7.1
  ERROR      /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js (undefined/us-east-1) 0.17s
      Error during integration test: Error: Command exited with status 1

Test Results:

Tests:    1 failed, 1 total
Error: Some integration tests failed!
    at main (/Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk/integ-runner/lib/cli.js:166:23)
error Command failed with exit code 1.

I'm not quite sure how this test is related or why it's failing now. Don't se anything related to denyAllIgwTraffic within that test or in any of your changes...

mtrspringer · 2024-08-08T18:48:27Z

@paulhcsun thanks yeah thats bizarre, i didnt make any changes to the nlb tests or source code. maybe something strange happened with the test assets during rebasing?

should i revert all asset/generated changes so that its just my desired code and let you run the suite again?

shikha372

Thank you @mtrspringer for submitting the PR, looking at the initial suggestion here I would also prefer a function based implementation that can give users the option to override only set of allowed values possible for alb-controller.

Pull request has been modified.

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

mtrspringer · 2024-10-31T14:42:57Z

@paulhcsun hi sorry i've been busy with my end-of-year work. i've just rebased the PR is there any chance you could run the integration tests again please? the iam-based disabling hack i commented about here has not worked as the aws-load-balancer-controller helm chart fails to complete the ingress update if the waf action it wants to perform fails. in this light i'd really like to try and get this feature over the line.

@shikha372 could you please elaborate on what you are looking for that's different from the current implementation? i have already updated to throw an error if a user attempts to set a helm chart value other that the AwsLoadBalancerController construct wants to set in response to previous comments. is this sufficient?

thanks

paulhcsun · 2024-11-05T18:40:22Z

Hi @mtrspringer, I've ran the integ tests and the integ.alb-controller.js test still fails with the error below. Strangely though a few other integ tests are failing too but that may because there are some changes on the branch that are out of date. Would you like me to push the snapshots for the integ.alb-controller.js test to help you debug the issue or is the error message enough to go off of?

Running test /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.alb-controller.js in us-east-1
/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/shared/base-load-balancer.js:97
                throw new Error(`'denyAllIgwTraffic' may only be set on load balancers with ${enums_1().IpAddressType.DUAL_STACK} addressing.`);
                ^

Error: 'denyAllIgwTraffic' may only be set on load balancers with dualstack addressing.
    at new BaseLoadBalancer (/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/shared/base-load-balancer.js:97:23)
    at new NetworkLoadBalancer (/Volumes/workplace/forks-cdk/mtrspringer/packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/nlb/network-load-balancer.js:165:9)
    at Object.<anonymous> (/Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js:13:1)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:142:12)
    at node:internal/main/run_main_module:28:49

This is the full list of tests that ran and failed:

Snapshot Results:

Tests:    6 failed, 1004 total
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.alb-controller.js
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/aws-elasticloadbalancingv2/test/integ.nlb-attributes.js
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/pipelines/test/integ.pipeline-security.js
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/pipelines/test/integ.pipeline-with-assets-single-upload.js
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/pipelines/test/integ.pipeline-with-assets.js
Failed: /Volumes/workplace/forks-cdk/mtrspringer/packages/@aws-cdk-testing/framework-integ/test/pipelines/test/integ.pipeline.js

paulhcsun · 2024-11-05T18:57:20Z

@shikha372 could you please elaborate on what you are looking for that's different from the current implementation? i have already updated to throw an error if a user attempts to set a helm chart value other that the AwsLoadBalancerController construct wants to set in response to previous comments. is this sufficient?

thanks

What @shikha372 is referring to is the solution that @TheRealAmazonKendra recommended to use a function that can be called after instantiation to override the desired values. I believe this is a good halfway compromise because we can still add in more methods for the properties that we want to support overrides for. I understand that for your use case and probably @Necrokefalos too, it would be fine to let you have control over all of the properties you want to override but this change would be applied to all other users too and we want to have some sort of safety net. In general in the CDK we try to make it impossible to set conflicting property values or properties that cannot be changed rather than to allow it to be set but prevent the deployment with error handling.

…ller

aws-cdk-automation · 2024-11-12T17:19:58Z

The pull request linter fails with the following errors:

❌ Features must contain a change to an integration test file and the resulting snapshot.

PRs must pass status checks before we can provide a meaningful review.

If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.

aws-cdk-automation · 2024-11-12T17:39:16Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 212f1ca
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mtrspringer · 2024-11-12T17:40:00Z

packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts

@@ -380,4 +405,22 @@ export class AlbController extends Construct {
    }
    return resources.map(rewriteResource);
  }
+
+  private validateHelmChartValues(values: {[key: string]: any}) {


@paulhcsun i have already implemented the validation as a function that restricts the keys the user is allowed to modify. do you mean that overriding the values should be an instance method? this does not make sense to me and does not align with how other properties of the AlbController HelmChart wrapper are set. for example, there is no albController.updateVersion() method, the helm chart version is passed in the constructor.

an instance method would also require updating the HelmChart construct to support modifying values after initialization (the AlbController constructor initializes the underlying HelmChart construct), which again makes the solution more complicated and expands beyond the scope of the AlbController - the HelmChart construct does not need to support value modification as it allows for passing arbitrary values already.

let me know if any modifications to this implementation are required and i'll be happy to make them.

mtrspringer · 2024-11-12T17:44:04Z

@paulhcsun thanks for the comments. im unsure what could be causing those errors - none of my test cases specify the denyAllIgwTraffic field. maybe it was caused by an underlying change? i can try to dig in more

i've responded to your comment about filtering the allowed values above, happy to hear your thoughts there.

github-actions bot added the beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK label Apr 4, 2024

aws-cdk-automation requested a review from a team April 4, 2024 11:47

github-actions bot added feature-request A feature should be added or improved. p2 labels Apr 4, 2024

aws-cdk-automation previously requested changes Apr 4, 2024

View reviewed changes

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Apr 5, 2024

mtrspringer force-pushed the aws-eks-alb-controller-values branch from ec6de8a to afcd343 Compare April 5, 2024 15:26

lpizzinidev approved these changes Apr 6, 2024

View reviewed changes

packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts Outdated Show resolved Hide resolved

packages/aws-cdk-lib/aws-eks/README.md Outdated Show resolved Hide resolved

aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Apr 6, 2024

github-actions bot mentioned this pull request Apr 8, 2024

Weekly PR metrics report #29758

Closed

mtrspringer force-pushed the aws-eks-alb-controller-values branch 3 times, most recently from 2926163 to 5e9a3f6 Compare April 16, 2024 09:36

paulhcsun self-assigned this Apr 18, 2024

paulhcsun previously requested changes Apr 19, 2024

View reviewed changes

github-actions bot mentioned this pull request May 1, 2024

Monthly PR metrics report #30026

Closed

mtrspringer force-pushed the aws-eks-alb-controller-values branch from 5e9a3f6 to 4289418 Compare May 13, 2024 16:44

mtrspringer force-pushed the aws-eks-alb-controller-values branch 2 times, most recently from 4625454 to b55781a Compare May 22, 2024 14:05

mtrspringer force-pushed the aws-eks-alb-controller-values branch from b55781a to cd2f5a8 Compare May 24, 2024 12:20

mtrspringer force-pushed the aws-eks-alb-controller-values branch from cd2f5a8 to 0d9c286 Compare June 7, 2024 15:10

mtrspringer force-pushed the aws-eks-alb-controller-values branch 3 times, most recently from bff33f3 to 285af63 Compare June 11, 2024 10:52

mtrspringer force-pushed the aws-eks-alb-controller-values branch 2 times, most recently from ab070af to 925b12e Compare July 17, 2024 14:09

mtrspringer force-pushed the aws-eks-alb-controller-values branch from 45a2cf9 to 6840f72 Compare August 13, 2024 16:14

shikha372 self-assigned this Sep 20, 2024

shikha372 previously requested changes Oct 1, 2024

View reviewed changes

mtrspringer force-pushed the aws-eks-alb-controller-values branch from 6840f72 to 89d6dbd Compare October 31, 2024 14:32

aws-cdk-automation requested changes Oct 31, 2024

View reviewed changes

mtrspringer force-pushed the aws-eks-alb-controller-values branch from 89d6dbd to 104f90c Compare October 31, 2024 14:36

mtrspringer force-pushed the aws-eks-alb-controller-values branch 3 times, most recently from f1af95a to c293db5 Compare November 4, 2024 12:14

mtrspringer added 4 commits November 12, 2024 17:15

add support for passing helm chart values to aws-load-balancer-contro…

248d8ae

…ller

add readme, integration test

1f3579d

fix test stack name

4045dab

changes from code review

212f1ca

mtrspringer force-pushed the aws-eks-alb-controller-values branch from b123336 to 212f1ca Compare November 12, 2024 17:17

mtrspringer commented Nov 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(eks): pass helm chart values to aws-load-balancer-controller #29723

feat(eks): pass helm chart values to aws-load-balancer-controller #29723

mtrspringer commented Apr 4, 2024

aws-cdk-automation left a comment

lpizzinidev left a comment

paulhcsun Apr 19, 2024 •

edited

Loading

mtrspringer Apr 26, 2024

paulhcsun Apr 19, 2024

mtrspringer Apr 26, 2024

paulhcsun Apr 26, 2024

mtrspringer May 13, 2024

paulhcsun May 14, 2024

paulhcsun May 17, 2024

paulhcsun May 21, 2024 •

edited

Loading

mtrspringer May 24, 2024

paulhcsun commented May 28, 2024

mtrspringer commented Jun 7, 2024

mtrspringer commented Jun 11, 2024

mtrspringer commented Jul 9, 2024

mtrspringer commented Jul 17, 2024

paulhcsun commented Aug 1, 2024

paulhcsun commented Aug 2, 2024

mtrspringer commented Aug 8, 2024

shikha372 left a comment •

edited

Loading

aws-cdk-automation left a comment

mtrspringer commented Oct 31, 2024

paulhcsun commented Nov 5, 2024

paulhcsun commented Nov 5, 2024

aws-cdk-automation commented Nov 12, 2024

aws-cdk-automation commented Nov 12, 2024

mtrspringer Nov 12, 2024

mtrspringer commented Nov 12, 2024

	readonly values?: {[key: string]: any};
	readonly helmChartValues?: {[key: string]: any};

feat(eks): pass helm chart values to aws-load-balancer-controller #29723

Are you sure you want to change the base?

feat(eks): pass helm chart values to aws-load-balancer-controller #29723

Conversation

mtrspringer commented Apr 4, 2024

Issue # (if applicable)

Reason for this change

Description of changes

Description of how you validated changes

Checklist

aws-cdk-automation left a comment

Choose a reason for hiding this comment

lpizzinidev left a comment

Choose a reason for hiding this comment

paulhcsun Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulhcsun May 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulhcsun commented May 28, 2024

mtrspringer commented Jun 7, 2024

mtrspringer commented Jun 11, 2024

mtrspringer commented Jul 9, 2024

mtrspringer commented Jul 17, 2024

paulhcsun commented Aug 1, 2024

paulhcsun commented Aug 2, 2024

mtrspringer commented Aug 8, 2024

shikha372 left a comment • edited Loading

Choose a reason for hiding this comment

aws-cdk-automation left a comment

Choose a reason for hiding this comment

mtrspringer commented Oct 31, 2024

paulhcsun commented Nov 5, 2024

paulhcsun commented Nov 5, 2024

aws-cdk-automation commented Nov 12, 2024

aws-cdk-automation commented Nov 12, 2024

AWS CodeBuild CI Report

Choose a reason for hiding this comment

mtrspringer commented Nov 12, 2024

paulhcsun Apr 19, 2024 •

edited

Loading

paulhcsun May 21, 2024 •

edited

Loading

shikha372 left a comment •

edited

Loading