Rotate container registry secret for Amazon's Elastic Container Registry (ECR)

[kylos101]

Is your feature request related to a problem? Please describe

This currently does not work per @mrzarquon

Describe the behaviour you'd like

Support ECR, but, persist image pull secrets in the cluster and rotate them on a schedule.

expand me to see alternative 2

One way to do it: - support accepting the name of an existing kube secrets in our install, instead of username/password (this is because if we are rotating secrets on a schedule, everytime the user does a kots deploy it will use whatever static value they entered for a secret) - take either aws keys OR a service account name that has permission to ECR (this means the user can create a service account in kuberenetes that has permissions to access ECR and never have to expose a secret to us) - setup a task that refreshes that docker secret with a new password every 12 hours - this sounds complicated but to people who use ecr, this is exactly what they have to do every time

Here are 33 lines of python for doing the password refresh from @mrzarquon experiment: https://github.com/mrzarquon/ecr-registry-refresh/blob/main/main.py#L47-L83

This'll be used by image-builder-mk3 (to create repos for base and workspaceImagel, so that it can build and push images) and well as registry-facade (so that it can pull the built images).

Describe alternatives you've considered

1. Provide native support to use ECR as a container registry and also allows the use of IAM Roles for Service Accounts (in pods). In other words, do not depend on a kubernetes secret referencing an image pull secret. It should work - for public and private ECR registries, too.
2. Continue not supporting ECR

Additional context

#2657 (originally mentioned)
#12060 (where we learned ECR is not supported, but lacks an issue, this one)

[lucasvaltl]

Note that we should not just support ECR, but also private ECR registries. I will update the description.

context from @mrzarquon about what would be needed here (copied from private slack message) - will also add this to the description:

but to support ECR requires us to :

• support accepting the name of an existing docker kube secrets in our install, instead of username/password (this is because if we are rotating secrets on a schedule, everytime the user does a kots deploy it will use whatever static value they entered for a secret)
• take either aws keys OR a service account name that has permission to ECR (this means the user can create a service account in kuberenetes that has permissions to access ECR and never have to expose a secret to us)
• setup a task that refreshes that docker secret with a new password every 12 hours
• this sounds complicated but to people who use ecr, this is exactly what they have to do every time

33 lines of python for doing the password refresh from my experiment: https://github.com/mrzarquon/ecr-registry-refresh/blob/main/main.py#L47-L83

[mrzarquon]

I created this issue around accepting references to secrets instead of insisting we store the secrets in kots configs - #13452

[atduarte]

Potential solutions discussed as part of the now closed #14891

[kylos101]

@atduarte @aledbf @lucasvaltl I updated the issue description to align with what was asked for in #14891, and moved the issue description that was originally here to the alternatives section. Prior to this change, they were not in fact duplicates - they were asking for different behaviors we'd like.

[atduarte]

@kylos101 why the title change? Think the previous one was more clear in regards to the desired outcome. 🤔

[kylos101]

@atduarte as part of the investigation, we found that there were two pieces of discrete work: (1) secret rotation and (2) creating the repos before pushing images to them. The former is being solved by @jenting in #15313. The latter will be solved by the Scout Team when they provision cells (hence closing). If we bump into something else, we'll likely create a separate issue, especially if we need to include someone else from the Workspace Team (aside from JenTing).