Failed to sign image with sigstore private key when using Harbor registry server

[STARRY-S]

Description

To reproduce:

1. Enable use-sigstore-attachments: true in /etc/containers/registries.d/default.yaml.
2. Copy container image with --sign-by-sigstore-private-key option to Harbor registry server.

[root@cbfe85fba3e6 /]# skopeo copy docker://docker.io/library/alpine:3 docker://harbor.xxx.com/library/alpine:3 --dest-tls-verify=false --sign-by-sigstore-private-key=./sigstore.private 
Passphrase for key ./sigstore.private: 
Getting image source signatures
Copying blob 4abcf2066143 skipped: already exists  
Copying config 05455a0888 done   | 
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
FATA[0005] copying system image from manifest list: writing signatures: reading manifest sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig in harbor.xxx.com/library/alpine: unknown: artifact library/alpine:sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig not found 

Debug output:

[root@cbfe85fba3e6 /]# skopeo copy docker://docker.io/library/alpine:3 docker://harbor.xxx.com/library/alpine:3 --dest-tls-verify=false --sign-by-sigstore-private-key=./sigstore.private --debug
Passphrase for key ./sigstore.private:
DEBU[0002] Using registries.d directory /etc/containers/registries.d
DEBU[0002] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0002] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0002] Found credentials for harbor.xxx.com/library/alpine in credential helper containers-auth.json in file /tmp/auth.json
DEBU[0002]  Lookaside configuration: using "default-docker" configuration
DEBU[0002]  No signature storage configuration found for harbor.xxx.com/library/alpine:3, using built-in default file:///var/lib/containers/sigstore
DEBU[0002] Looking for TLS certificates and private keys in /etc/docker/certs.d/harbor.xxx.com
DEBU[0002]  Sigstore attachments: using "default-docker" configuration
DEBU[0002] Using registries.d directory /etc/containers/registries.d
DEBU[0002] Trying to access "docker.io/library/alpine:3"
DEBU[0002] No credentials matching docker.io/library/alpine found in /tmp/auth.json
DEBU[0002] No credentials for docker.io/library/alpine found
DEBU[0002]  Lookaside configuration: using "default-docker" configuration
DEBU[0002]  No signature storage configuration found for docker.io/library/alpine:3, using built-in default file:///var/lib/containers/sigstore
DEBU[0002] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0002]  Sigstore attachments: using "default-docker" configuration
DEBU[0002] GET https://registry-1.docker.io/v2/
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0002] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io
DEBU[0003] GET https://registry-1.docker.io/v2/library/alpine/manifests/3
DEBU[0003] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0003] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0004] Source is a manifest list; copying (only) instance sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 for current system
DEBU[0004] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0
DEBU[0004] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0004] IsRunningImageAllowed for image docker:docker.io/library/alpine:3
DEBU[0004]  Using default policy section
DEBU[0004]  Requirement 0: allowed
DEBU[0004] Overall: allowed
Getting image source signatures
DEBU[0004] Reading /var/lib/containers/sigstore/library/alpine@sha256=6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0/signature-1
DEBU[0004] Looking for sigstore attachments in docker.io/library/alpine:sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig
DEBU[0004] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig
DEBU[0004] Content-Type from manifest GET is "application/json"
DEBU[0004] Fetching sigstore attachment manifest failed, assuming it does not exist: reading manifest sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig in docker.io/library/alpine: manifest unknown
DEBU[0004] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0004] ... will first try using the original manifest unmodified
DEBU[0004] Checking if we can reuse blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0004] Checking /v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
DEBU[0004] GET https://harbor.xxx.com/v2/
DEBU[0004] Ping https://harbor.xxx.com/v2/ err Get "https://harbor.xxx.com/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://harbor.xxx.com/v2/", Err:(*errors.errorString)(0x5616c1ab7820)})
DEBU[0004] GET http://harbor.xxx.com/v2/
DEBU[0004] Ping http://harbor.xxx.com/v2/ status 401
DEBU[0004] GET http://harbor.xxx.com/service/token?account=admin&scope=repository%3Alibrary%2Falpine%3Apull%2Cpush&service=harbor-registry
DEBU[0004] HEAD http://harbor.xxx.com/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
DEBU[0004] ... already exists
DEBU[0004] Skipping blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8 (already present):
Copying blob 4abcf2066143 skipped: already exists
DEBU[0004] Downloading /v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
DEBU[0004] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
Copying config 05455a0888 [--------------------------------------] 0.0b / 1.4KiB | 0.0 b/s
DEBU[0004] No compression detected
DEBU[0004] Compression change for blob sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0004] Using original blob without modification
DEBU[0004] Checking /v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
DEBU[0004] HEAD http://harbor.xxx.com/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
Copying config 05455a0888 done   |
Writing manifest to image destination
DEBU[0004] PUT http://harbor.xxx.com/v2/library/alpine/manifests/3
Creating signature: Signing image using a sigstore signature
Storing signatures
DEBU[0004] Looking for sigstore attachments in harbor.xxx.com/library/alpine:sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig
DEBU[0004] GET http://harbor.xxx.com/v2/library/alpine/manifests/sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig
DEBU[0004] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0004] Fetching sigstore attachment manifest failed: reading manifest sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig in harbor.xxx.com/library/alpine: unknown: artifact library/alpine:sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig not found
FATA[0004] copying system image from manifest list: writing signatures: reading manifest sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig in harbor.xxx.com/library/alpine: unknown: artifact library/alpine:sha256-6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0.sig not found

• skopeo version 1.15.0
• Harbor version v2.8.5

---

Related code position: https://github.com/containers/image/blob/v5.30.0/docker/docker_client.go#L1112C3-L1115C4

[STARRY-S]

Original issue: containers/image#2203
This issue has not been updated recently. Is there any progress on it?

[mtrmac]

Thanks for reaching out.

If it is going to be fixed, it needs to be fixed in the c/image issue, not here; and that requires capturing a full sample HTTP response.

[STARRY-S]

Thanks, I'll close this issue.