OIDC Provider ARN not accepted as Federated Principal for iam-role-for-service-accounts-eks

[webdog]

Description

When using this module paired with the EKS Module, I am passing the OIDC Provider ARN from the EKS module output module.eks.oidc_provider_arn to this module.

I am returning the following error from AWS:

│ Error: failed creating IAM Role (tutorialcluster): MalformedPolicyDocument: Federated principals must be valid domain names or SAML metadata ARNs
│       status code: 400, request id: 2a43ba14-04fb-4cfe-8192-14eb57df28fe
│ 
│   with module.iam_eks_role.aws_iam_role.this[0],
│   on .terraform/modules/iam_eks_role/modules/iam-role-for-service-accounts-eks/main.tf line 25, in resource "aws_iam_role" "this":
│   25: resource "aws_iam_role" "this" {
│ 
╵

It appears that in the dynamic statement block:

terraform-aws-iam/modules/iam-role-for-service-accounts-eks/main.tf

Lines 4 to 23 in 66c80ff

	dynamic "statement" {
	for_each = var.oidc_providers
	content {
	effect = "Allow"
	actions = ["sts:AssumeRoleWithWebIdentity"]
	principals {
	type = "Federated"
	identifiers = [statement.value.provider_arn]
	}
	condition {
	test = "StringEquals"
	variable = "${replace(statement.value.provider_arn, "/^(.*provider/)/", "")}:sub"
	values = [for sa in statement.value.namespace_service_accounts : "system:serviceaccount:${sa}"]
	}
	}
	}
	}

...It is unpacking the OIDC Providers map and using the OIDC provider ARN to specify a federated principal. I'm not sure how to proceed, since this ARN is generated by the EKS module. (Line 13 in the above block)

Versions

• Terraform: 1.1.5

• Provider(s):
  provider registry.terraform.io/hashicorp/aws v3.74.0

• provider registry.terraform.io/hashicorp/cloudinit v2.2.0
• provider registry.terraform.io/hashicorp/hcp v0.22.0
• provider registry.terraform.io/hashicorp/local v2.1.0
• provider registry.terraform.io/hashicorp/null v3.1.0
• provider registry.terraform.io/hashicorp/tls v3.1.0

• Module: 4.13.2

Reproduction

Steps to reproduce the behavior:

Code Snippet to Reproduce

module "iam_eks_role" {
  source = "registry.terraform.io/terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = lower(var.cluster_info.name)

  oidc_providers = {
    one =  {
      provider_arn = module.eks.oidc_provider_arn
      namespace_service_accounts = [ "default:tutorial" ]
    }
  }
}

Expected behavior

It should accept the OIDC Provider ARN I assume?

Actual behavior

Error message above.

Terminal Output Screenshot(s)

See above.

Additional context

None. Thanks!

[bryantbiggs]

do you have a full configuration reproduction? what EKS module version are you using and is the OIDC provider enabled on the cluster?

[webdog]

AFAICT yes, the OIDC Provider is enabled on the cluster. The EKS module sets enable_irsa = true which then creates the OIDC Provider:

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/6fe818d0a350c6eb8b65cb461fe5e480597f2ffe/variables.tf#L244-L248

The cluster returns an OIDC Issuer URL:

aws eks describe-cluster --name tutorialCluster --query "cluster.identity.oidc.issuer"

"https://oidc.eks.us-east-1.amazonaws.com/id/<ID>"

As I've looked into this, I think this could be an issue with the EKS module, as the module.eks.oidc_provider_arn is returning an empty string when passed to the IAM role being created by this module. (Output as provider_arn in this snippet), and causing Principal.Federated to be empty as well.

Outputs:

consul_cert_data = <sensitive>
eks_cluster_host = "https://<ID>.gr7.us-east-1.eks.amazonaws.com"
eks_data = <sensitive>
provider_arn = ""
vault_data = <sensitive>

  # module.iam_eks_role.aws_iam_role.this[0] will be created
  + resource "aws_iam_role" "this" {

. . .
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRoleWithWebIdentity"
                      + Condition = {
                          + StringEquals = {
                              + :sub = "system:serviceaccount:default"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + Federated = ""
                        }
    }

. . . 

[bryantbiggs]

it is hard to tell where your issue lies without seeing a reproduction of your configuration (EKS and IAM). I can tell you that these examples work however https://github.com/terraform-aws-modules/terraform-aws-iam/blob/master/examples/iam-role-for-service-accounts-eks/main.tf

[webdog]

Apologies, since it seems like the EKS module was returning the empty string, wasn't sure if you still needed the configuration information 😄

But here's how I have configured both modules (Including the version for EKS as you requested)

EKS

# Deploys Amazon EKS
module "eks" {
  # Full URL due to this issue: https://github.com/VladRassokhin/intellij-hcl/issues/365
  source                               = "registry.terraform.io/terraform-aws-modules/eks/aws"
  version                              = "18.3.0"
  cluster_name                   = var.cluster_info.name
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true


  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {}
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
    }
  }

  vpc_id = module.aws_vpc.vpc_id
  subnet_ids = setunion(
    module.aws_vpc.public_subnets,
    module.aws_vpc.private_subnets
  )

  # Node Groups
  eks_managed_node_group_defaults = {
    ami_type               = var.node_group_configuration.ami_type
    disk_size              = var.node_group_configuration.disk_size_gigs
    instance_types         = var.node_group_configuration.instance_types
    vpc_security_group_ids = [module.aws.aws_security_group_id, module.aws.aws_hashicups_sg]
  }

  eks_managed_node_groups = {
    blue = {}
    green = {
      min_size     = 2
      max_size     = 2
      desired_size = 2
    }
  }
  tags = {
    Environment = "dev"
  }

}

And iam-role-for-service-accounts-eks:

module "iam_eks_role" {
  source = "registry.terraform.io/terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  role_name = lower(var.cluster_info.name)

  oidc_providers = {
    one =  {
      provider_arn = module.eks.oidc_provider_arn
      namespace_service_accounts = [ "default:tutorial" ] # this originally was set to "${var.namespace}:${var.service_account}", but exchanged those values for the current string.
    }
  }
}

I also have trace logs from terraform plan and apply. They contain sensitive data, so if there's a way I can securely send that to you, that would be great.

[bryantbiggs]

perfect, so now its clear. You are using v18.3.0 but the provider_arn was not added until v18.6.0 terraform-aws-modules/terraform-aws-eks#1870

this is why the version is pinned to at minimum v18.6.0 in the examples

terraform-aws-iam/examples/iam-role-for-service-accounts-eks/main.tf

Line 219 in f1fecd9

version = "~> 18.6"

[webdog]

Oh yikes! Thank you for your patience, and the quick follow-up. Changing that now 😸

[bryantbiggs]

I was wrong above - oidc_provider was added in v18.6.0, oidc_provider_arn should be there.

However, do try the latest version and let me know if there is any change

[webdog]

I bumped up to v18.6.0 and that populated the string for the federated principal, and the targeted apply now completes.

(From my output):

    "provider_arn": {
      "value": "arn:aws:iam::ACCT_ID:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/OIDC_ID",
      "type": "string"

I was wrong above - oidc_provider was added in v18.6.0, oidc_provider_arn should be there.

For some reason oidc_provider_arn is available when using 18.3, but it was returning an empty value. Moving to 18.6 seems to have resolved the issue. I can report that to the EKS module folks in any case.

[bryantbiggs]

ah I see now, in v18.4.0 we opted to enable IRSA by default terraform-aws-modules/terraform-aws-eks#1849

you could stay at v18.3.0 if desired by setting enable_irsa = true

If you can though, I would go all the way up to the latest v18.9.0; otherwise users who use self-managed node groups will see some warnings due to the v4 AWS provider

[webdog]

Just wanted to provide a quick follow up; I bumped to 18.9.0 and was able to successfully map the IAM role to the Service Account I created in k8s. Thanks for the help!

[bryantbiggs]

awesome, glad it worked out!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.