Harbor to support cross-repository mount

[adriannieto-attechnest]

As per https://github.com/opencontainers/distribution-spec/blob/main/spec.md#mounting-a-blob-from-another-repository Harbor should return a 202 (https://docs.docker.com/registry/spec/api/#cross-repository-blob-mount)

Crane (but potentially other libraries) has implemented this feature which is causing tools relying on cross-repository mounts to not work anymore to with Harbor (google/go-containerregistry#1393 google/go-containerregistry#1388)

This issue is happening in the very latest Harbor version.

Versions: Harbor 2.5.1, Helm chart: 1.9.1
Additional context:

[jonjohnsonjr]

To add a little context, when attempting a cross-repository mount from a non-existent source repository (the from parameter), harbor fails and returns a 401 with UNAUTHORIZED: project examplegroup not found: project examplegroup not found, whereas it should return a 202 with the Location to perform the upload.

I'm not familiar with the harbor codebase but this comment seems to indicate that it wasn't clear what to do in this case, but I believe the spec has been clarified now? cc @reasonerjt

harbor/src/server/middleware/artifactinfo/artifact_info.go

Lines 83 to 84 in 44f477e

	// Fail early for now, though in docker registry an invalid may return 202
	// it's not clear in OCI spec how to handle invalid from parm

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[adriannieto-attechnest]

This issue is still happening, harbor is not following registry spec

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[adriannieto-attechnest]

Still happening

[wy65701436]

thanks @adriannieto-attechnest I will take time to investigate it, Harbor should align the distrubtion spec.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[zyyw]

@jonjohnsonjr @adriannieto-attechnest , currently Harbor does support cross-repository blob mounting within the same registry, verified with Harbor 2.8.2 & crane 0.15.2, distribution spec.

ubuntu@shengwen:~/Misc/crane-hub/crane-0.15.2$ ./crane copy HARBOR_IP/library/nginx:1.23.3 HARBOR_IP/project15/nginx:1.23.3
2023/07/06 09:38:26 Copying from HARBOR_IP/library/nginx:1.23.3 to HARBOR_IP/project15/nginx:1.23.3
2023/07/06 09:38:27 mounted blob: sha256:258f176fd22609802ed5def498ce4c7d1fa80a8ddba258b6e9eca0364df3ddb3
2023/07/06 09:38:27 mounted blob: sha256:a0bc35e7077319c74cd587e1c911894c0bc003e1783c0bc2e59e351d41cfa543
2023/07/06 09:38:27 mounted blob: sha256:077b9569ff86b205b47334ce8fa1c4bcf55b124b2fe880f40b56f9932f856f68
2023/07/06 09:38:27 mounted blob: sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa
2023/07/06 09:38:27 mounted blob: sha256:3082a16f3b619613a40d0b9056c6142493980314187ef6b6281909a40f6b7147
2023/07/06 09:38:27 mounted blob: sha256:7e9b29976cce358fbdd777b91ce2c52d18090dd1507d0616793e9ce8cc15979c
2023/07/06 09:38:27 mounted blob: sha256:3f8a00f137a0d2c8a2163a09901e28e2471999fde4efc2f9570b91f1c30acf94
2023/07/06 09:38:27 HARBOR_IP/project15/nginx:1.23.3: digest: sha256:7f797701ded5055676d656f11071f84e2888548a2e7ed12a4977c28ef6114b17 size: 1570
ubuntu@shengwen:~/Misc/crane-hub/crane-0.15.2$

And Harbor (v2.8.2) doesn't officially support cross-registry blob mounting or cross-origin blob mounting, which means mounting blob between two Harbor registry endpoint is currently not officially supported in Harbor (v2.8.2). And this behavior of cross-registry blob mounting is not defined in distribution spec yet.

ubuntu@shengwen:~/Misc/crane-hub/crane-0.15.2$ ./crane copy HARBOR_1_IP/library/nginx:1.23.3 HARBOR_2_IP/registry2-project15-1/nginx:1.23.3

And it does return 401 when doing the POST, for example, proxy.log from HARBOR_2_IP machine

Jul  6 09:43:58 shengwen proxy[962]: HARBOR_2_IP - "POST /v2/registry2-project15-1/nginx/blobs/uploads/?from=library%2Fnginx&mount=sha256%3A3f8a00f137a0d2c8a2163a09901e28e2471999fde4efc2f9570b91f1c30acf94&origin=HARBOR_1_IP HTTP/1.1" 401 176 "-" "crane/0.15.2 go-containerregistry/0.15.2" 0.005 0.004 .

I'm thinking enabling cross-registry blob mounting in Harbor after the distribution spec of this part is finalized.
cc: @wy65701436 How do you think of it?

[zyyw]

As for this error:

UNAUTHORIZED: project examplegroup not found: project examplegroup not found

This is because Harbor wraps the NOT_FOUND error to be UNAUTHORIZED with the middleware here:

harbor/src/server/middleware/v2auth/auth.go

Line 167 in fb52fdb

lib_http.SendError(rw, errors.UnauthorizedError(err).WithMessage(err.Error()))

This is not only the way for POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>, but also the current behavior for some other APIs. This behavior does be different from distribution spec. And we can track it in this issue:

• Referrers API returns response code different from spec #18418

[jessehu]

hi, is this issue fixed by which PR?

[MinerYang]

Hi all,
Here's some add-on info about this issue

• distribution-spec#mounting-a-blob-from-another-repository only claim cross-repository mount with same registry. so what @adriannieto-attechnest mentioned in this crane-issue-1393 is out of discussion scope.
• What distribution-spec claims is Alternatively, if a registry does not support cross-repository mounting or is unable to mount the requested blob, it SHOULD return a 202.. Combine with distribution-spec conformance test, there's several cases that would return 202.

1. registry dose not support cross-repository. -> Not applicable to Harbor since we support it.
2. Unable to mount blob due to non exist blob in the /registry/v2/repositories/<name>/_layers/sha256, harbor would return 202 for this POST request itself

"POST /v2/test01/hello-world/blobs/uploads/?from=test%2Fhello-world&mount=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e&origin=<xx.xxx.xxx.xx> HTTP/1.1" 202 0 "-" "crane/0.19.1 go-containerregistry/0.19.1" 0.012 0.012 .

3. Unable to mount blob due to query without from request but with mount

"POST /v2/test/hello-world/blobs/uploads/?mount=sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 202 0 "-" "curl/7.81.0" 0.017 0.017 .

4. Unable to mount blob due to non exist repository, e.g. library/hello which repository hello is not exist.

"POST /v2/test/hello-world/blobs/uploads/?from=library/hello&mount=sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 202 0 "-" "curl/7.81.0" 0.015 0.015 .

All above situations return 202 from distribution directly, harbor did nothing to intercept the response.

• As for push to/from a non-exist project return 401 error with project xxx not found it is checked by design to confirm if source or target project is exist. Image cannot push to harbor before project is created. https://goharbor.io/docs/2.10.0/working-with-projects/create-projects/.
  This is designed for authentication, however, current v2 API have not taken care of this part.
  Legacy API spec do have failure auth check