Manage EKS system:masters members somehow

[tgeoghegan]

@branlwyd and I learned last week that the AWS IAM user that creates an EKS cluster is automatically and invisibly added to a system:masters Kubernetes group, allowing management of Kubernetes cluster resources. However, other IAM users get no privileges, even if they have administrator privileges at the AWS layer, and must be explicitly added to the kube-system/aws-auth config map in an appropriate mapUsers section1.

We should work out a way to manage this access via Terraform.

[tgeoghegan]

The difficulty is that I'd prefer not to enumerate our AWS IAM user ARNs in .tfvars files that get checked into a public repository. Perhaps we could create an IAM role like k8s-cluster-admin and grant it system:master via mapRoles, and then permit IAM users to impersonate that role, at the cost of (1) manually managing membership in the role, and (2) requiring users to use an AWS profile/kubectl configuration that assumes a role instead of performing operations as their IAM user.

[tgeoghegan]

• update the prio-server bringup docs in https://github.com/abetterinternet/docs to reflect any changes made to address this issue

[tgeoghegan]

I discussed what I want to do about this here, but I've run into snags while trying to implement it. Introducing readonly and admin IAM roles and using them in the aws-auth config map works, but the problem is doing that via Terraform. The aws-auth map gets created automatically by EKS, and the Kubernetes provider for Terraform doesn't provide a way to add values to an existing config map, so the only way to do this in Terraform would be to do something complicated with a local-exec provisioner, which I would really rather not do. AWS have an open issue about this, or rather the general problem of programmatic management of EKS cluster access. Once they provide API for that, and it gets exposed through the aws TF provider, we should adopt it. In the meantime, given how rarely we provision clusters, I think we will just document the need to manually fix up the kube-system/aws-auth config map, though we can still automate the creation of the AWS IAM roles and the k8s ClusterRole and ClusterRoleBinding.

[tgeoghegan]

The prio-server onboarding docs discuss managing the config map. As I noted, I don't think we can do much better until there is EKS API plumbed up through Terraform for us. In the meantime, I'm taking this out of the current tech debt milestone.