Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regression: v1.1.0 doesn't work with GCP artifact registry using custom docker image , v1.0.0 correctly works #1204

Closed
1 task
gpongelli opened this issue Dec 14, 2023 · 11 comments
Labels
bug Something isn't working duplicate This issue or pull request already exists question Further information is requested

Comments

@gpongelli
Copy link

What happened in your environment?

the error happens with ORAS 1.1.0 :

root@f54f52505a49:/# oras pull -d <GCP-artifact-registry-path-toartifact>:<version>
DEBU[0000] Request #0
> Request URL: "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/manifests/<version>"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.1.0"
DEBU[0000] Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Content-Type": "application/json; charset=utf-8"
   "X-Xss-Protection": "0"
   "Content-Length": "102"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Www-Authenticate": "Bearer realm=\"https://europe-docker.pkg.dev/v2/token\",service=\"europe-docker.pkg.dev\",scope=\"repository:<GCP-artifact-registry-path-toartifact>:pull\""
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "Date": "Thu, 14 Dec 2023 10:26:51 GMT"
DEBU[0000] Request #1
> Request URL: "https://europe-docker.pkg.dev/v2/token?scope=repository%3A<GCP-artifact-registry-path-toartifact>%3Apull&service=europe-docker.pkg.dev"
> Request method: "GET"
> Request headers:
   "User-Agent": "oras/1.1.0"
DEBU[0000] Response #1
< Response Status: "403 Forbidden"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Thu, 14 Dec 2023 10:26:51 GMT"
   "Content-Length": "226"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/json; charset=utf-8"
   Error: failed to resolve 1.8.3.0-linux-x64: GET "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/manifests/<version>": GET "https://europe-docker.pkg.dev/v2/token?scope=repository%3A<GCP-artifact-registry-path-toartifact>%3Apull&service=europe-docker.pkg.dev": response status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/<GCP-artifact-registry-path-toartifact>" (or it may not exist)
root@04f2353edd9a:/# oras version
Version:        1.1.0
Go version:     go1.21.0
Git commit:     7079c468a06fb5815c99395eb4aaf46dd459d3fa
Git tree state: clean

obviously some sensitive information are masked

What did you expect to happen?

previous ORAS 1.0.0 works as expected:

root@4f8d3f2ee0e0:/# oras pull -d <GCP-artifact-registry-path-toartifact>:<version>
DEBU[0000] Request #0
> Request URL: "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/manifests/<version>"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.0.0"
DEBU[0000] Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Www-Authenticate": "Bearer realm=\"https://europe-docker.pkg.dev/v2/token\",service=\"europe-docker.pkg.dev\",scope=\"repository:<GCP-artifact-registry-path-toartifact>:pull\""
   "X-Content-Type-Options": "nosniff"
   "X-Xss-Protection": "0"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Frame-Options": "SAMEORIGIN"
   "Date": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Content-Length": "102"
DEBU[0000] Request #1
> Request URL: "https://europe-docker.pkg.dev/v2/token?scope=repository%3A<GCP-artifact-registry-path-toartifact>%3Apull&service=europe-docker.pkg.dev"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
DEBU[0000] Response #1
< Response Status: "200 OK"
< Response headers:
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/json"
   "X-Content-Type-Options": "nosniff"
DEBU[0000] Request #2
> Request URL: "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/manifests/<version>"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
DEBU[0000] Response #2
< Response Status: "200 OK"
< Response headers:
   "Content-Length": "683"
   "Content-Type": "application/vnd.oci.image.manifest.v1+json"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
   "X-Xss-Protection": "0"
   "Docker-Content-Digest": "sha256:4bab09616371cd2dbcbace686e44a19f7714dc06424f4a2d2cebd62804be9cf6"
   "X-Frame-Options": "SAMEORIGIN"
   "Date": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
Downloading f148b4f8d409 linux-x64
DEBU[0000] Request #3
> Request URL: "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/blobs/sha256:<artifact-hash>"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
DEBU[0000] Response #3
< Response Status: "302 Found"
< Response headers:
   "Content-Type": "text/html; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Location": "/artifacts-downloads/namespaces/<GCP-artifact-registry-path-toartifact>/downloads/<very-long-string>"
   "X-Content-Type-Options": "nosniff"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Date": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
DEBU[0000] Request #4
> Request URL: "https://europe-docker.pkg.dev/artifacts-downloads/namespaces/<GCP-artifact-registry-path-toartifact>/downloads/<very-long-string>"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
   "Referer": "https://europe-docker.pkg.dev/v2/<GCP-artifact-registry-path-toartifact>/blobs/sha256:<artifact-hash>"
DEBU[0000] Response #4
< Response Status: "200 OK"
< Response headers:
   "Content-Length": "234224"
   "Date": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Expires": "Thu, 14 Dec 2023 11:06:35 GMT"
   "Cache-Control": "private, max-age=0"
   "X-Goog-Hash": "crc32c=evV6gA=="
   "Content-Type": "application/octet-stream"
   "Accept-Ranges": "bytes"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "X-Guploader-Uploadid": "ABPtcPpMqTQGvdkPXB76fMfTLFPCMTKVl-48qs_TNEfWmLlqaunE-aVdR5ympk5xQ4FOH5e6Jg"
   "Server": "UploadServer"
Downloaded  f148b4f8d409 linux-x64
Pulled [registry] <GCP-artifact-registry-path-toartifact>:<version>
Digest: sha256:4bab09616371cd2dbcbace686e44a19f7714dc06424f4a2d2cebd62804be9cf6
root@4f8d3f2ee0e0:/# oras version
Version:        1.0.0
Go version:     go1.20.2
Git commit:     b58e7b910ca556973d111e9bd734a71baef03db2
Git tree state: clean
root@4f8d3f2ee0e0:/#

obviously some sensitive information are masked

How can we reproduce it?

I've created two custom docker images with ORAS manually installed, one with ORAS v1.0.0 and another with ORAS v1.1.0, to be run through jenkins on a k8s environment, this is the dockerfile used to create the images:

# escape=`
ARG BUILD_IMAGE=gcr.io/cloud-builders/docker  # ubuntu 20.04.6 LTS based image

# google cloud builder with oras

FROM $BUILD_IMAGE

ARG ORAS_VERSION

# install oras tool
RUN mkdir -p /tmp/oras-install/ && `
    curl -LO "https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_${ORAS_VERSION}_linux_amd64.tar.gz" && `
    ls -la . && `
    tar -zxf ./oras_${ORAS_VERSION}_linux_amd64.tar.gz -C /tmp/oras-install/  && `
    cp /tmp/oras-install/oras /usr/bin/  && `
    cp /tmp/oras-install/oras /usr/sbin/  && `
    chmod a+x /usr/bin/oras /usr/sbin/oras && `
    rm -rf ./oras_${ORAS_VERSION}_linux_amd64.tar.gz /tmp/oras-install/

ENTRYPOINT [ "/bin/bash" ]

then use a .docker/config.json file with the content of a k8s service account used to push/pull artifact into GCP artifact registry.

The two docker images are executed, mounting .docker folder to the container's user .docker folder to do authentication, with:
docker run -it -v C:\Users\gpongelli\.docker\:/root/.docker <custom-docker-image>:1.1.0
docker run -it -v C:\Users\gpongelli\.docker\:/root/.docker <custom-docker-image>:1.0.0

into those images I've executed the oras pull command reported above.

Thanks!

What is the version of your ORAS CLI?

Version: 1.1.0
Go version: go1.21.0
Git commit: 7079c46
Git tree state: clean

What is your OS environment?

docker image

Are you willing to submit PRs to fix it?

  • Yes, I am willing to fix it.
@gpongelli gpongelli added bug Something isn't working triage New issues or PRs to be acknowledged by maintainers labels Dec 14, 2023
@qweeah qweeah added the question Further information is requested label Dec 15, 2023
@qweeah
Copy link
Contributor

qweeah commented Dec 15, 2023

From the given logs, I can see that

  1. both v1.1.0 and v1.0.0 are requesting token from the same token server https://europe-docker.pkg.dev/v2/token?scope=repository%3A<GCP-artifact-registry-path-toartifact>%3Apull&service=europe-docker.pkg.dev
  2. v1.1.0 request got rejected with 403.
    It's possible that v1.1.0 failed to pickup the credential but we need more user input to confirm it.

@gpongelli Are you running both commands in the same container? What base image are you using to run commands? How is the token provided(have you run oras login before?

@gpongelli
Copy link
Author

Hi @qweeah ,
there exist two different docker images, and therefore two docker containers; both images are made from the same dockerfile attached:

  • for 1.1.0 image the build command is docker build --build-arg ORAS_VERSION=1.1.0 -t docker-oras:1.1.0 .
  • for 1.0.0 image the build command is docker build --build-arg ORAS_VERSION=1.0.0 -t docker-oras:1.0.0 .

those images run with root user, and the run command are
docker run -it -v C:\Users\gpongelli\.docker\:/root/.docker docker-oras:1.1.0
docker run -it -v C:\Users\gpongelli\.docker\:/root/.docker docker-oras:1.0.0

into C:\Users\gpongelli\.docker\ folder, passed to the container to do authentication as described here, there is a config.json with following content:

{
    "auths": 
        {"https://europe-docker.pkg.dev": 
            {"username": "_json_key", 
            "password": "{\n  \"type\": \"service_account\",\n  \"project_id\": \"<GCP-PROJECT>\",\n  \"private_key_id\": \"<KEY-ID>\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n<PRIVATE-KEY>\\n-----END PRIVATE KEY-----\\n\",\n  \"client_email\": \"<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com\",\n  \"client_id\": \"<CLIENT-ID>\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com\"\n}", "email": "<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com", "auth": "<AUTHENTICATION-VERY-LONG-FIELD>"
        }
    }
}

So, I'm using:

  • same /root/.docker/config.json file into container used by ORAS
  • same artifact in GCP to be downloaded
    happens that ORAS 1.0.0 correctly download the artifact with only one command oras pull -d <GCP-artifact-registry-path-toartifact>:<version> while ORAS 1.1.0 does NOT download the artifact with the SAME command.

oras login is NEVER executed, not in ORAS 1.0.0 nor in ORAS 1.1.0 , is it needed/introduced in v1.1.0 ?

@gpongelli
Copy link
Author

I'm trying to do oras login running the docker-oras:1.1.0 image locally, it asks me Username and Token and it's something I don't want because this image has to be run in a jenkins pipeline without user intervention, that's why I pass the config.json for the service account.
image

@qweeah
Copy link
Contributor

qweeah commented Dec 16, 2023

@gpongelli Thanks for the input. The only ambiguity I can see in the auth file, is that both auth and password field are provided.

Can you help try v1.0.1? It uses same cred SDK(oras-credentials-go v0.2.0) as ORAS v1.1.0.

You can verify it easily via using oras CLI container, e.g.

docker run -v ~/.docker/config.json:/root/.docker/config.json ghcr.io/oras-project/oras:v1.0.1 pull -d <GCP-artifact-registry-path-toartifact>:<version>

If it fails, can you help remove the auth field and try again?

@gpongelli
Copy link
Author

@gpongelli Thanks for the input. The only ambiguity I can see in the auth file, is that both auth and password field are provided.

So which is the content that oras needs?
Anyway, having more fields than needed should not be a problem, if they’re ignored.

Can you help try v1.0.1? It uses same cred SDK(oras-credentials-go v0.2.0) as ORAS v1.1.0.

You can verify it easily via using oras CLI container, e.g.

docker run -v ~/.docker/config.json:/root/.docker/config.json ghcr.io/oras-project/oras:v1.0.1 pull -d <GCP-artifact-registry-path-toartifact>:<version>

If it fails, can you help remove the auth field and try again?

Removing the auth field means having an empty dictionary into config.json, is this what you mean?

I’ll let you know.

@qweeah
Copy link
Contributor

qweeah commented Dec 18, 2023

Removing the auth field means having an empty dictionary into config.json, is this what you mean?

No, I mean try with below config.json file:

{
    "auths": 
        {"https://europe-docker.pkg.dev": 
            {"username": "_json_key", 
            "password": "{\n  \"type\": \"service_account\",\n  \"project_id\": \"<GCP-PROJECT>\",\n  \"private_key_id\": \"<KEY-ID>\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\n<PRIVATE-KEY>\\n-----END PRIVATE KEY-----\\n\",\n  \"client_email\": \"<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com\",\n  \"client_id\": \"<CLIENT-ID>\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com\"\n}", "email": "<SERVICE-ACCOUNT-EMAIL>.iam.gserviceaccount.com"
        }
    }
}

The "auths"."https://europe-docker.pkg.dev"."auth" should be removed if pulling with v1.0.1 fails.

@gpongelli
Copy link
Author

Hi @qweeah ,
those are the results:

  • ORAS 1.0.1 with the same config.json file does NOT work, same error
  • ORAS 1.0.1 with changed config.json file (without "auths"."https://europe-docker.pkg.dev"."auth" ) does NOT work, same error

@qweeah
Copy link
Contributor

qweeah commented Dec 19, 2023

@Wwwsylvia @wangxiaoxuan273 Any idea on why oras-credentials-go v0.2.0 might fail to pick up the credential in config.json?

@qweeah
Copy link
Contributor

qweeah commented Dec 19, 2023

@gpongelli There is an unexpected prefix of the auths, the https://
This issue is fixed in #1143 which will be released in v1.2.0.

If you want to use oras v1.1.0, you need replace "https://europe-docker.pkg.dev" with "europe-docker.pkg.dev".

@qweeah qweeah added duplicate This issue or pull request already exists and removed triage New issues or PRs to be acknowledged by maintainers labels Dec 19, 2023
@qweeah
Copy link
Contributor

qweeah commented Dec 19, 2023

@gpongelli To check if #1143 fixed your issue or not, you may try with our ghcr build of main branch:

docker run -v ~/.docker/config.json:/root/.docker/config.json ghcr.io/oras-project/oras:main pull -d <GCP-artifact-registry-path-toartifact>:<version>

@gpongelli
Copy link
Author

@gpongelli There is an unexpected prefix of the auths, the https:// This issue is fixed in #1143 which will be released in v1.2.0.

If you want to use oras v1.1.0, you need replace "https://europe-docker.pkg.dev" with "europe-docker.pkg.dev".

Hi @qweeah ,
I confirm that this way, removing https://, works with ORAS 1.1.0 .
It works also having "auths"."europe-docker.pkg.dev"."auth" key into config.json file.

Thank you very much!

@qweeah qweeah closed this as completed Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working duplicate This issue or pull request already exists question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants