Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

sebschlue · 2023-05-11T08:23:01Z

Is there an existing issue for this?

I have searched the existing issues

What did you expect to happen?

Image pull from registry.k8s.io should work but I simply get 403 Forbidden

Debugging Information

Normal Pulling 87s (x4 over 2m58s) kubelet Pulling image "registry.k8s.io/metrics-server/metrics-server:v0.6.3"
Warning Failed 86s (x4 over 2m57s) kubelet Failed to pull image "registry.k8s.io/metrics-server/metrics-server:v0.6.3": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/metrics-server/metrics-server:v0.6.3": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.k8s.io/v2/metrics-server/metrics-server/blobs/sha256:817bbe3f2e5179b5fa7dd4085c87aab1faaf9dc879e3b1d05796901a006f58ab: 403 Forbidden

Anything else?

No response

Code of Conduct

I agree to follow this project's Code of Conduct

ameukam · 2023-05-11T08:40:59Z

@sebschlue Do you mind go through the debugging guide so we can get more informations about this issue you are facing ?

What you provided is enough for us to properly investigate this issue.

sebschlue · 2023-05-11T09:41:30Z

I will try some tests.

First test:

$ kubectl run curl --rm -it --image=curlimages/curl --restart=Never --command -- curl -v https://registry.k8s.io/v2/
*   Trying 34.96.108.209:443...
* Connected to registry.k8s.io (34.96.108.209) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=registry.k8s.io
*  start date: May  9 06:25:11 2023 GMT
*  expire date: Aug  7 07:21:05 2023 GMT
*  subjectAltName: host "registry.k8s.io" matched cert's "registry.k8s.io"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /v2/]
* h2h3 [:scheme: https]
* h2h3 [:authority: registry.k8s.io]
* h2h3 [user-agent: curl/8.0.1-DEV]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7ff772e90a90)
> GET /v2/ HTTP/2
> Host: registry.k8s.io
> user-agent: curl/8.0.1-DEV
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< docker-distribution-api-version: registry/2.0
< x-cloud-trace-context: 928d497daa6bd00cb18f27f7dfc768c9
< date: Thu, 11 May 2023 09:23:38 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 0
< via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
* Connection #0 to host registry.k8s.io left intact
pod "curl" deleted````

sebschlue · 2023-05-11T09:43:36Z

Second test:

$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane ls registry.k8s.io/metrics-server
dev
v0.2.1
v0.3.3
v0.3.4
v0.3.5
v0.3.6
pod "crane" deleted

sebschlue · 2023-05-11T09:49:01Z

third test:

$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane pull --verbose registry.k8s.io/metrics-server:v0.3.6 metrics-server-v0.3.6.tgz
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/crane, falling back to streaming logs:
2023/05/11 09:46:49 --> GET https://registry.k8s.io/v2/
2023/05/11 09:46:49 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip


2023/05/11 09:46:49 <-- 200 https://registry.k8s.io/v2/ (79.136354ms)
2023/05/11 09:46:49 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 11 May 2023 09:46:49 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: aed4ad49eb58ccf06be191eafb4e8199
Content-Length: 0


2023/05/11 09:46:49 --> GET https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
2023/05/11 09:46:49 GET /v2/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip


2023/05/11 09:46:49 <-- 307 https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6 (20.375601ms)
2023/05/11 09:46:49 HTTP/2.0 307 Temporary Redirect
Content-Length: 132
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 09:46:49 GMT
Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: fff6db6c3bb348da31246ec53538b46a

<a href="https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6">Temporary Redirect</a>.


2023/05/11 09:46:49 --> GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
2023/05/11 09:46:49 GET /v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: europe-west4-docker.pkg.dev
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
Accept-Encoding: gzip


2023/05/11 09:46:49 <-- 200 https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 (108.131929ms)
2023/05/11 09:46:49 HTTP/2.0 200 OK
Content-Length: 738
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Thu, 11 May 2023 09:46:49 GMT
Docker-Content-Digest: sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1784,
      "digest": "sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 654467,
         "digest": "sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9885841,
         "digest": "sha256:b2f4b24bed0dade029d7164c181e0c8b2c0846fb3a1e2aabf4141480382849b2"
      }
   ]
}
Error: saving tarball metrics-server-v0.3.6.tgz: open metrics-server-v0.3.6.tgz: permission denied
pod "crane" deleted
pod vault/crane terminated (Error)```

TerryHowe · 2023-05-11T14:46:13Z

Last message just shows permission denied for writing the tarball. Did you get past that?

sebschlue · 2023-05-11T15:04:59Z

Yes :) Now you can see the 403 at the end:

$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane pull --verbose registry.k8s.io/metrics-server:v0.3.6 /tmp/metrics-server-v0.3.6.tgz
2023/05/11 15:03:21 --> GET https://registry.k8s.io/v2/
2023/05/11 15:03:21 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip


2023/05/11 15:03:21 <-- 200 https://registry.k8s.io/v2/ (56.484713ms)
2023/05/11 15:03:21 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 11 May 2023 15:03:21 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 1a61c1a94b76867c3e81ec2fb1e68981
Content-Length: 0


2023/05/11 15:03:21 --> GET https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
2023/05/11 15:03:21 GET /v2/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip


2023/05/11 15:03:21 <-- 307 https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6 (20.880947ms)
2023/05/11 15:03:21 HTTP/2.0 307 Temporary Redirect
Content-Length: 132
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 15:03:21 GMT
Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 67ede95221b1d2fa5bdf3816b9233c3d

<a href="https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6">Temporary Redirect</a>.


2023/05/11 15:03:21 --> GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
2023/05/11 15:03:21 GET /v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: europe-west4-docker.pkg.dev
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
Accept-Encoding: gzip


2023/05/11 15:03:22 <-- 200 https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 (102.210221ms)
2023/05/11 15:03:22 HTTP/2.0 200 OK
Content-Length: 738
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Thu, 11 May 2023 15:03:21 GMT
Docker-Content-Digest: sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1784,
      "digest": "sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 654467,
         "digest": "sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9885841,
         "digest": "sha256:b2f4b24bed0dade029d7164c181e0c8b2c0846fb3a1e2aabf4141480382849b2"
      }
   ]
}
2023/05/11 15:03:22 --> GET https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
2023/05/11 15:03:22 GET /v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip


2023/05/11 15:03:22 <-- 307 https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b (88.617019ms)
2023/05/11 15:03:22 HTTP/2.0 307 Temporary Redirect
Content-Length: 207
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 15:03:22 GMT
Location: https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 2cd3976c07d6d9a3e676c04a710dce09

<a href="https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b">Temporary Redirect</a>.


2023/05/11 15:03:22 --> GET https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
2023/05/11 15:03:22 GET /containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b HTTP/1.1
Host: prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Referer: https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
Accept-Encoding: gzip


2023/05/11 15:03:22 <-- 403 https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b (36.755538ms)
2023/05/11 15:03:22 HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 11 May 2023 15:03:21 GMT
Server: AmazonS3
X-Amz-Id-2: Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=
X-Amz-Request-Id: T4K2XR8208VF2NTX

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>T4K2XR8208VF2NTX</RequestId><HostId>Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=</HostId></Error>
0


Error: saving tarball /tmp/metrics-server-v0.3.6.tgz: unable to calculate manifest: GET https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b: unexpected status code 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>T4K2XR8208VF2NTX</RequestId><HostId>Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=</HostId></Error>

BenTheElder · 2023-05-11T15:29:57Z

This works from my machine:

curl https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b

So it's not the blob at least, something between S3 and your host.

Is it possible your environment has some restrictions on S3 access? Previously kubernetes/k8s.io#4214 (comment)

BenTheElder · 2023-05-11T15:31:15Z

https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html

Perhaps https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html#troubleshoot-403-orgs

sebschlue · 2023-05-12T06:17:59Z

Thanks a lot, will check it :)

sebschlue · 2023-05-14T10:31:22Z

Okay, I did not yet figure out the root cause, but it's now clear that it is not a general issue but limited to my environment, so ticket can be closed. Thanks a lot for your help :)

sebschlue added kind/support Categorizes issue or PR as a support question. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels May 11, 2023

sebschlue closed this as completed May 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

sebschlue commented May 11, 2023

ameukam commented May 11, 2023

sebschlue commented May 11, 2023

sebschlue commented May 11, 2023

sebschlue commented May 11, 2023

TerryHowe commented May 11, 2023

sebschlue commented May 11, 2023

BenTheElder commented May 11, 2023

BenTheElder commented May 11, 2023

sebschlue commented May 12, 2023

sebschlue commented May 14, 2023

Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

Comments

sebschlue commented May 11, 2023

Is there an existing issue for this?

What did you expect to happen?

Debugging Information

Anything else?

Code of Conduct

ameukam commented May 11, 2023

sebschlue commented May 11, 2023

sebschlue commented May 11, 2023

sebschlue commented May 11, 2023

TerryHowe commented May 11, 2023

sebschlue commented May 11, 2023

BenTheElder commented May 11, 2023

BenTheElder commented May 11, 2023

sebschlue commented May 12, 2023

sebschlue commented May 14, 2023