Access RDS databases via EKS with helm install

[stevenGravy]

This steps through installing a helm chart with the required role to enable access to a RDS postgresql database via Teleport. This approach should work for other similar helm installs with similar role requirements.

Prerequisites:

• EKS Kubernetes cluster with OIDC provider
• RDS database with Password and IAM database authentication enabled.
• RDS URI (e.g. postgres-aurora.cmkm1qh.us-east-1.rds.amazonaws.com:5432) is available from the K8s cluster.
• jq is used below to help collect info
• aws CLI tool

This assumes you have created users in the database such as the below for postgresql that can access via IAM.

CREATE USER alice;
GRANT rds_iam TO alice;

Setup your role to enable access

Create your policy

Collect the region, AWS account and resource id. This policy will be used to allow the Teleport pods to access the RDS database.

From the database access guide:
The database resource ID is shown on the Configuration tab of a particular database instance in the RDS control panel, under "Resource id". For regular RDS database it starts with db- prefix. For Aurora, use the database cluster resource ID (cluster-), not the individual instance ID.

# sub your values
REGION=us-east-1
AWS_ACCOUNT=1234567890
RESOURCE_ID=cluster-12345
cat > rds.policy << EOF
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:${REGION}:${AWS_ACCOUNT}:dbuser:${RESOURCE_ID}/*"
         ]
      }
   ]
}
EOF

Create the policy

aws iam create-policy --policy-name my-rds-policy --policy-document file://rds.policy

The my-rds-policy will be attached to your role.

Collect your oidc issuer

CLUSTER_NAME=mycluster
REGION=us-east-1
aws eks describe-cluster --name ${CLUSTER_NAME} --region ${REGION} | jq .cluster.identity.oidc.issuer
# "https://oidc.eks.us-east-1.amazonaws.com/id/44969AD34F07D62C3BE6ACCF41B4"

Collect the last string (44969AD34F07D62C3BE6ACCF41B4 as above).

Create a role

Create the trust policy

The trust policy is used to match and allow assuming role with the web identity.

REGION=us-east-1
AWS_ACCOUNT=1234567890
OIDC_PROVIDER=44969AD34F07D62C3BE6ACCF41B4
cat > trustpolicy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${OIDC_PROVIDER}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF

Create the role

Create a role with the trust policy and attach policy to access db.

AWS_ACCOUNT=1234567890
aws iam create-role --role-name my-rds-role --assume-role-policy-document file://trustpolicy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::${AWS_ACCOUNT}:policy/my-rds-policy --role-name my-rds-role

Install helm chart

Create a database token

tctl tokens add --type=db --format=text

Replace the values and create the values.yaml file.

TOKEN=123456
PROXY_ADDRESS=teleport.example.com:443
RDS_URI=postgres-aurora-instance-1.cmzxvm1qh.us-east-1.rds.amazonaws.com:5432
REGION=us-east-1
AWS_ACCOUNT=123456789
ROLE=my-rds-role
cat > values.yaml << EOF
authToken: ${TOKEN}
proxyAddr: ${PROXY_ADDRESS}
roles: db
databases:
- name: example
  uri: "${RDS_URI}"
  protocol: "postgres"
  static_labels:
    env: dev
  aws:
    region: "${REGION}"
annotations:
  serviceAccount:
    eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT}:role/${ROLE}
EOF

Install the helm chart to the Kubernetes cluster

helm repo add teleport https://charts.releases.teleport.dev
helm repo update

Install the agent

helm install teleport-agent teleport/teleport-kube-agent -f values.yaml --create-namespace --namespace teleport-agent

Confirm connection

Confirm that the database is listed and can be connected

Create user

tctl users add --roles=access alice --db-users='*' --db-names='*'

Connect

tsh login --proxy=teleport.example.com --auth=local --user=alice
tsh db connect --db-user=alice --db-name=postgres example

Troubleshooting

Here are typical things to check:

• the RDS database has IAM authentication enabled
• the uri port is available from the Kubernetes cluster.
• Check the logs for errors
  kubectl logs -f -n teleport-agent pods/teleport-agent-0

[GavinFrazar]

I think the IAM trust policy should have a subject condition as well

https://aws.github.io/aws-eks-best-practices/security/docs/iam/#scope-the-iam-role-trust-policy-for-irsa-roles-to-the-service-account-name-namespace-and-cluster