Add support for mapping a registry to a specific AWS_PROFILE

[micahhausler]

In the EKS Distro, we use ecr-credential-helper with buildkit to pull base images and push images, but we often have base images in one account, and need to push images to a separate account. Rather than using ECR repository policies to grant a single IAM role permission to do a in-account pull and a cross-account push, we use have a separate role our build job can assume to push cross-account to ECR public. I know ECR Public is already requested (#248), but simply adding support won't solve our use case.

ECR public doesn't (yet?) support repository policies, and we want our builder process to use a separate IAM role for ECR private pulls than the one used to push to ECR public.

Our current docker config looks like the following:

{
    "credHelpers": {
        "111122223333.dkr.ecr.us-west-2.amazonaws.com": "ecr-login",
        "222233334444.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
    }
}

because using {"credsStore": "ecr-login"} swallows any token you try to save to the config file.

Right now, we have a gross hack that saves the ecr public credential in the docker config.

# Token expires after 12 hours
TOKEN=$(aws --region us-east-1 ecr-public get-authorization-token \
    --output=text \
    --query 'authorizationData.authorizationToken')

DOCKER_CONFIG=${DOCKER_CONFIG:-"~/.docker"}
mkdir -p $DOCKER_CONFIG
if [ ! -f $DOCKER_CONFIG/config.json ]; then
    echo '{}' > $DOCKER_CONFIG/config.json
fi
# Gross, abominable hack until ecr-credential-helper supports public ECR
# See https://github.com/awslabs/amazon-ecr-credential-helper/issues/248
cat $DOCKER_CONFIG/config.json | \
    jq --arg TOKEN $TOKEN '.auths."public.ecr.aws".auth = $TOKEN' > $DOCKER_CONFIG/config.json.new
cp $DOCKER_CONFIG/config.json.new $DOCKER_CONFIG/config.json

This allows buildkit to pull from one account and push from another in the same command.

cat << EOF > ./Dockerfile
ARG BASE_IMAGE
FROM $BASE_IMAGE
# Do stuff
EOF

BASE_IMAGE="111122223333.dkr.ecr.us-west-2.amazonaws.com/something/something:tag"
IMAGE="public.ecr.aws/eks-distro/something/something:tag"
buildctl \
	build \
	--frontend dockerfile.v0 \
	--opt platform=linux/amd64,linux/arm64 \
	--opt build-arg:BASE_IMAGE=$BASE_IMAGE \
	--local dockerfile=./ \
	--local context=. \
	--output type=image,oci-mediatypes=true,name=$IMAGE,push=true

I'd be great to have a config file that could associate registries with specific AWS profiles. For example, if my ~/.aws/config looked like this

[default]
output = json
region = us-west-2
role_arn=$AWS_ROLE_ARN
web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile publish]
role_arn = arn:aws:iam::777788889999:role/EcrPublicPublishRole
region = us-east-1
source_profile=default

I would want a way to associate 111122223333.dkr.ecr.us-west-2.amazonaws.com with AWS_PROFILE=default and public.ecr.aws with AWS_PROFILE=publish.

This is not a formal proposal for a config spec, but it could be something like:

registryConfigs:
  # implicit, since the helper would use the `AWS_PROFILE` of whatever was invoking docker/ecr-credential-helper
  #111122223333.dkr.ecr.us-west-2.amazonaws.com:
  #  aws_profile: "default"
  public.ecr.aws:
    aws_profile: "publish"

which would allow me to just set my Docker config to {"credsStore": "ecr-login"} not think about it

[micahhausler]

This might also solve #247, I'll let @johnkeates weigh in if he wants to fold his ask into this issue

[johnkeates]

@micahhausler This would indeed be the same issue. I was under the impression that AWS_PROFILE was only used/read if we also specifically set AWS_SDK_LOAD_CONFIG to true, so we'd still need both environment variables, right?

[micahhausler]

@micahhausler This would indeed be the same issue. I was under the impression that AWS_PROFILE was only used/read if we also specifically set AWS_SDK_LOAD_CONFIG to true, so we'd still need both environment variables, right?

AWS_PROFILE is used by newer non-python SDKs (like Go) and and AWS CLI v2, and AWS_DEFAULT_PROFILE is used by the CLI v1 and in the Go SDK if AWS_SDK_LOAD_CONFIG=1 is set.

[TBBle]

AWS_SDK_LOAD_CONFIG is effectively set by default, as of #201 (merged in February 2020), which will be part of the next release.

[johnkeates]

Odd, I couldn't get it to work without that. Perhaps I was using an old release.

[TBBle]

There hasn't been a release with this fix yet, it happened after 0.4.0, which is the most recent release. It'll be in the next release.

[johnkeates]

Ah yes, that makes sense. I was confusing a merge with a direct release afterwards ;-)

[samuelkarp]

@TBBle @johnkeates Version 0.5.0 now loads the shared config by default so you no longer need to set AWS_SDK_LOAD_CONFIG=1.

[dekimsey]

We use this credential helper to publish resources across organizational boundaries which have unique IAM credentials (for reasons). Being able to associate a given registry with a specific AWS profile would address some real pain points.

In particular, manipulating images into the usgov partition is really hard right now since we cannot associate an AWS profile with a given registry.

[gunzy83]

Why is this labeled more info needed? I think this is an obvious use case particularly if you want your users to use assumed roles with least privilege.

In our case we have our users aws sso login with the default profile once a day and then the commands they run from there (via code etc) use a number of profiles mapped to roles that allow those commands to go through and not much more. It also allows for our CI system to assume those roles so the code works the same on every machine (minor differences in the credential source in ~/.aws/config). This would be really helpful to remove yet another command users have to run. I have upvoted in the OP as well.

[dudicoco]

any updates on this one?
just started to use this awesome tool and was surprised to find out that this is not supported out of the box.

[fagiani]

Also, how can one set the priority or specify that regardless of having ~/.aws/config or ~/.aws/credentials or even AWS_ACCESS_KEY_ID, use the IAM role for the container running for example on CodeBuild?

If I set any of those, and trigger any further docker commands, docker login will repeat and prioritize them even if before I've cached a initial request with the IAM ROLE.

[TBBle]

@fagiani That might be better as a separate feature request ticket, it's not really related to this one. I suspect it's also something that will be driven by the underlying SDK, not amazon-ecr-credential-helper, so it might not be an easy thing to fix (or alternatively, there might be an env-var or something that already does this, kind-of the inverse of AWS_SDK_LOAD_CONFIG)

[fagiani]

@TBBle that makes sense. I'll investigate the issue further and if required open an issue on the SDK project. Sorry for the noise here.

Keep Rocking!

[oferziss-armis]

we are trying to achieve pulling our images from an ECR repository located on the aws partition from an EKS cluster located on the aws-gov partition.

due to how the integration between eks and ECR works, we resolved to using the creds helper to provide the node dynamic credentials for the non-gov account.
unfortunately, when i provide the kubelet with the AWS_PROFILE to use we are running into multiple issues:

1. if we add the env var to kubelet prior to running the eks-bootstrap.sh script it then fails to register the node to the cluster. same if we define the creadentials to be in the default profile
2. if we change the env var post bootstrap and reload the kubelet, we are then unable to fetch the aws_node and cni images as we are trying to fetch an aws-gov image using aws credentials.

binding a registry to an aws profile will solve all of these rather cleanly and will allow us to avoid doing blasphemous stuff like injecting a cronjob into the node to use awscli to login into the registry all the time...

this requirement is a real pain point and will honestly be a great solve if we can get it.....

[dougbyrne]

As a workaround, I've been adding bash scripts in my $PATH that look like this:

#!/bin/bash

AWS_PROFILE=my-cool-profile docker-credential-ecr-login "$@"

That will be saved as /usr/local/bin/docker-credential-ecr-login-my-cool-profile and then I can add ecr-login-my-cool-profile as the name of the credential helper for that repo. It would be nice to have some kind of config mapping for this instead.

[purajit]

Have there been any further internal discussion on this? We're still using hacks to get around this, but it's particularly bothersome when you have to distribute the hack to an entire org.

[callumcrossley12]

I've raised a PR and given it a go. My changes work well for me (testing locally) but will continue to use over the next week or so. PR is #894 and feedback would be appreciated.