Error fetching image manifest list from private AWS ECR repo

[AvivBenchorin]

I have been trying to use enroot import with a private AWS ECR repository, and have been running into an issue when retrieving the image manifest list. I was able to setup my .credentials file and successfully complete the authentication step following NVIDIA/enroot#59, however when fetching the image manifest list I got the following error:

$ enroot import docker://0000.dkr.ecr.us-east-2.amazonaws.com/image-repo:image-tag
[INFO] Querying registry for permission grant
[INFO] Authenticating with user: AWS
[INFO] Using credentials from file: /path/to/enroot/.credentials
[INFO] Authentication succeeded
[INFO] Fetching image manifest list
[ERROR] Could not process JSON input

This error is being thrown by the common::jq function being called in docker::_download, and when printing the curl response that is being passed into common::jq, I found that the response to the manifest list request was Not Authorized in plain text. This error has been thrown both when installing enroot from source and from packages.

Do you have any insight into resolving this issue?

Enroot version: 3.4.0
Docker version: 20.10.17

[AvivBenchorin]

In the same environment in which I saw the error above, I was able to successfully pull the image from the private AWS ECR repo using docker pull with the following commands:

# Authentication
aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 0000.dkr.ecr.us-east-2.amazonaws.com

# Pulling
docker pull 0000.dkr.ecr.us-east-2.amazonaws.com/image-repo:image-tag

So I do not think that the error is from having inadequate permissions or authentication issues in my environment.

[3XX0]

How did you setup the credentials file? Did you follow https://github.com/NVIDIA/enroot/blob/master/doc/cmd/import.md#description?

[AvivBenchorin]

Yes, my .credentials file is:
machine 0000.dkr.ecr.us-east-2.amazonaws.com login AWS password $(aws ecr get-login-password --region us-east-2)
And is located in the directory which my enroot.conf file set as the ENROOT_CONFIG_PATH.

[3XX0]

Um not sure what's happening then, maybe ECR changed their authentication scheme.
Authentication succeeded means that you successfully retrieved a token though, so not sure what changed there.
You got try to use mitmproxy to spot the difference between Enroot and Docker in the specific case.

[ricktbaker]

Was there a solution to this problem? Currently running into the same issue. I get a proper authentication, but end up with the same unable to parse json input error.

[lipovsek-aws]

I'm using this workaround:

ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 123.dkr.ecr.us-east-2.amazonaws.com
docker pull 123.dkr.ecr.us-east-2.amazonaws.com/my-image
enroot import --output /fsx/containers/my-image.sqsh dockerd://123.dkr.ecr.us-east-2.amazonaws.com/my-image

I think the underlying issue is still there.

[jlbutler]

@3XX0 hello! Apologies I meant to help look at this a long while back, and got distracted with comparing how it might have worked before. I did look at the temporary patch in #59 which I agree is a bit gross, but does work.

ECR doesn't have a auth endpoint. It's designed to be a multi-tenant registry, and treats repos as a tenancy boundary. It can be fiddly to deal with when working with the v2 OCI API. The GetAuthorizationToken API needs to be used, or 'aws ecr get-authorization-token' vs 'aws ecr get-login-password'.

So, the best way I think is to add in a change like you had done in that issue comment. I can file a PR tomorrow AM (a bit late here now) now, PTAL and let me know what you think. Thanks!