Allow deployment on AWS EKS using Service Account & IRSA

[andrea-defraia]

I've deployed Tapir on AWS EKS using Terraform.
Assuming you have a cluster, this is the needed code (might be useful to add this as an example)

The below code will work on a "standard" cluster, but, the service account gets ignored.
To be clear: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

The service account is not used, the pod has hardcoded a call to the EKS node IAM role.
Adding the needed permissions to the node IAM role allows the deployment to run without issues, but I think it should be allowed/encouraged to use IRSA instead.

Providers:

  region = "eu-west-1"
}

resource "random_pet" "pet" {
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

data "aws_eks_cluster" "eks" {
  name = module.eks.name
}

provider "kubernetes" {
config_path = "./my-kubeconfig-file"
}

Tapir:

resource "kubernetes_namespace_v1" "tapir" {
  metadata {
    name = "tapir"
    labels = merge(
      module.labels.labels,
      { name = "tapir" }
    )
  }
  lifecycle {
    ignore_changes = [metadata[0].annotations]
  }
}

resource "kubernetes_deployment_v1" "tapir" {
  metadata {
    namespace = kubernetes_namespace_v1.tapir.id
    name      = "tapir"
    labels = merge(
      module.labels.labels,
      { name = "tapir" }
    )
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.tapir.arn
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        name = "tapir"
      }
    }

    template {
      metadata {
        labels = {
          name = "tapir"
        }
        annotations = {
          "eks.amazonaws.com/role-arn" = aws_iam_role.tapir.arn
        }
      }

      spec {
        service_account_name = "tapir"
        container {
          image = "pacovk/tapir"
          name  = "tapir"

          port {
            container_port = 8080
          }

          env {
            name  = "S3_STORAGE_BUCKET_NAME"
            value = module.tapir_s3.id
          }
          env {
            name  = "S3_STORAGE_BUCKET_REGION"
            value = data.aws_region.current.name
          }
          env {
            name  = "REGISTRY_HOSTNAME"
            value = "tapir.<mydomain>"
          }
          env {
            name  = "REGISTRY_PORT"
            value = 443
          }
        }
      }
    }
  }
}

resource "kubernetes_service_v1" "tapir" {
  metadata {
    name      = "tapir"
    namespace = kubernetes_namespace_v1.tapir.id
  }
  spec {
    selector = {
      name = "tapir"
    }
    port {
      port        = 443
      target_port = 8080
    }

    type = "ClusterIP"
  }
}

resource "kubernetes_ingress_v1" "tapir" {
  metadata {
    name      = "tapir"
    namespace = kubernetes_namespace_v1.tapir.id
    labels = {
      service = "tapir"
    }
    # Values here:https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/
    annotations = {
      "kubernetes.io/ingress.class"                        = "alb"
      "alb.ingress.kubernetes.io/group.name"               = "default"
      "alb.ingress.kubernetes.io/scheme"                   = "internet-facing"
      "alb.ingress.kubernetes.io/backend-protocol"         = "HTTPS"
      "alb.ingress.kubernetes.io/target-type"              = "ip"
      "alb.ingress.kubernetes.io/listen-ports"             = "[{\"HTTPS\": 443}]"
      "alb.ingress.kubernetes.io/ssl-policy"               = "ELBSecurityPolicy-TLS-1-2-Ext-2018-06"
      "alb.ingress.kubernetes.io/certificate-arn"          = module.cert.arn
    }
  }
  spec {
    rule {
      host = "tapir.<mydomain>"
      http {
        path {
          path      = "/*"
          path_type = "ImplementationSpecific"
          backend {
            service {
              name = "tapir"
              port {
                number = 443
              }
            }
          }
        }
      }
    }
  }
}
resource "kubernetes_service_account_v1" "tapir" {
  metadata {
    name      = "tapir"
    namespace = kubernetes_namespace_v1.tapir.id
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.tapir.arn
    }
  }
  automount_service_account_token = false
}

resource "kubernetes_cluster_role" "tapir" {
  metadata {
    name = "tapir"
  }

  rule {
    api_groups = [""]
    resources  = ["services", "endpoints", "pods"]
    verbs      = ["get", "watch", "list"]
  }

  rule {
    api_groups = ["extensions", "networking.k8s.io"]
    resources  = ["ingresses"]
    verbs      = ["get", "watch", "list"]
  }

  rule {
    api_groups = [""]
    resources  = ["nodes"]
    verbs      = ["list", "watch"]
  }
}

resource "kubernetes_cluster_role_binding" "tapir" {
  metadata {
    name = "tapir"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.tapir.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = "tapir"
    namespace = kubernetes_namespace_v1.tapir.id
  }
}


resource "aws_iam_role" "tapir" {
  name               = "tapir"
  description        = "Role assumed by EKS ServiceAccount tapir"
  assume_role_policy = data.aws_iam_policy_document.tapir_sa.json
  tags = merge(
    module.labels.labels,
    {
      "Resource" = "aws_iam_role.tapir"
    }
  )
}

data "aws_iam_policy_document" "tapir_sa" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [
        "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(module.eks.cluster_oidc_issuer_url, "https://", "")}"
      ]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(module.eks.cluster_oidc_issuer_url, "https://", "")}:sub"
      values   = ["system:serviceaccount:${kubernetes_namespace_v1.tapir.id}:tapir"]
    }
  }
}

resource "aws_iam_role_policy" "tapir" {
  role   = aws_iam_role.tapir.id
  policy = data.aws_iam_policy_document.tapir.json
}

data "aws_iam_policy_document" "tapir" {
  statement {
    sid    = "S3Access"
    effect = "Allow"
    resources = [
      "${module.tapir_s3.arn}/*",
      module.tapir_s3.arn
    ] #tfsec:ignore:aws-iam-no-policy-wildcards
    actions = [
      "s3:Describe*",
      "s3:List*",
      "s3:Get*",
      "s3:Put*"
    ]
  }

  statement {
    sid    = "DynamoDbAccess"
    effect = "Allow"
    resources = [
      "arn:aws:dynamodb:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/Modules",
      "arn:aws:dynamodb:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/Providers",
      "arn:aws:dynamodb:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/Reports"
    ]
    actions = [
      "dynamodb:DescribeLimits",
      "dynamodb:DescribeTimeToLive",
      "dynamodb:ListTagsOfResource",
      "dynamodb:DescribeReservedCapacityOfferings",
      "dynamodb:DescribeReservedCapacity",
      "dynamodb:ListTables",
      "dynamodb:BatchGetItem",
      "dynamodb:BatchWriteItem",
      "dynamodb:CreateTable",
      "dynamodb:DeleteItem",
      "dynamodb:GetItem",
      "dynamodb:GetRecords",
      "dynamodb:PutItem",
      "dynamodb:Query",
      "dynamodb:UpdateItem",
      "dynamodb:Scan",
      "dynamodb:DescribeTable"
    ]
  }

  statement {
    sid       = "KMSAccess"
    effect    = "Allow"
    resources = [module.kms.arn]
    actions = [
      "kms:Decrypt",
      "kms:Describe*",
      "kms:Encrypt",
      "kms:GenerateDataKey*",
      "kms:List*",
      "kms:ReEncrypt*"
    ]
  }
}


module "tapir_s3" {
  source        = "../../../modules/s3"
  labels        = module.labels.labels
  kms_key       = module.kms.arn
  name          = "${random_pet.pet.id}-tapir"
  force_destroy = true
}

Tapir pod logs:
2023-09-04 09:59:56,270 INFO [io.quarkus] (main) Installed features: [amazon-dynamodb, amazon-s3, cdi, config-yaml, elasticsearch-rest-client, hibernate-validator, quinoa, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-openapi, vertx] 2023-09-04 09:59:56,275 INFO [cor.Bootstrap] (main) Validate GPG key configuration provided 2023-09-04 09:59:56,277 INFO [cor.Bootstrap] (main) Start to bootstrap registry database [dynamodb] 2023-09-04 09:59:56,884 WARN [sof.ama.aws.aut.cre.int.WebIdentityCredentialsUtils] (main) To use web identity tokens, the 'sts' service module must be on the class path. 2023-09-04 09:59:57,329 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): software.amazon.awssdk.services.dynamodb.model.DynamoDbException: User: arn:aws:sts::123456789101:assumed-role/safe-drake-test-cluster20230904091517423300000010/i-0bde33149ce682d09 is not authorized to perform: dynamodb:CreateTable on resource: arn:aws:dynamodb:eu-west-1:123456789101:table/Modules because no identity-based policy allows the dynamodb:CreateTable action (Service: DynamoDb, Status Code: 400, Request ID: QQP3NPAQJ9U1UJILGST8IMFDVVVV4KQNSO5AEMVJF66Q9ASUAAJG)

User: arn:aws:sts::123456789101:assumed-role/safe-drake-test-cluster20230904091517423300000010/i-0bde33149ce682d09 is not authorized to perform...
Pod should instead be using the Service Account IAM Role specified in Annotations: "eks.amazonaws.com/role-arn" = aws_iam_role.tapir.arn

[PacoVK]

Thanks for this comprehensive report and the code example.
There are two things:

• The extension for STS needs to be added, it just makes sense to restrict to IRSA. The fallback will be the Node role
• I really like to add the K8s part as an example, based on the code here

I'll try to provide both these days

[PacoVK]

Released 0.4.0 which enables IRSA capabilities for EKS deployments