Registry Lifecycle Management: Current State

[SteveLasker]

On the July 14, 2021 OCI call, we discussed inventorying each registries implementation of lifecycle management in the hopes of creating some standards and minimal standard set of expectations for users and tools to work across registries.

In an attempt to use discussions:
Each registry service/project owns a single response to this heading. As more info surfaces, edit your section.

[SteveLasker]

Azure Container Registry (ACR)

draft to get the conversation started

• ACR Docs

ACR provides a means to delete tags, and/or digests. If you delete a tag, the digests remain in the registry, with the ability to delete untagged manifests.

• What about deleting an index? Indexes can be deleted (by tag or digest), but the underlying manifests it references are maintained.
• Can you delete a manifest that's referenced by an index? (hmmm, I need to check what we do)

[brackendawson]

IBM Cloud Container Registry (ICR)

There are a number of lifecycle management options in ICR available either through distribution which I will call the native API or through IBM's own api which I will call the extended API.

• Anything in the distribution 2.7 native API.
• Can delete image via the CLI/extended API. This removes manifest and all tags which reference it from a repository.
• Can untag which removes a tag link but leaves the manifest in place. This is available via the CLI/extended API and via the native API: Distribution has almost all of the code needed to allow DELETE requests to /v2/{repo}/tag/{tag} but doesn't expose it, ICR does, we are keen to contribute this change to distribution.
• Can remove all untagged manifests in a repository via the CLI/extended API.
• Can set retention policies per namespace (first "folder" of repository) that can keep only the most recently pushed n tags in a repository and optionally keep or discard untagged manifests in a repository. This is in the CLI/extended API.

On top the above options for deleting content, all images/tags deleted by users and retention policies are actually moved to trash and held there for 30 days. Users can restore content from trash to the original repository overwriting anything that might now be there.

• Can delete namespace (first "folder" of repository) via the CLI/extended API which removes all content in that repository and the repositories under it. This is also triggered if the cloud account which owns the namespace is deleted for any reason.
• Blobs which are not referenced by a manifest (excluding manifest list) will be deleted asynchronously.

ICR does allow you to delete a manifest which is referenced from a manifest-list, subsequent pulls through the list fail gracefully in the client and it looks just like if you were pulling a manifest which no longer exists:

% docker pull uk.icr.io/alan/list
Using default tag: latest
latest: Pulling from alan/list
manifest for uk.icr.io/alan/list:latest not found: manifest unknown: manifest unknown

I suppose this could be confusing depending on how your extended API surfaces manifest-lists.

It can also be more confusing because if a list is pushed in a different repository to the manifest/s which is references; then distribution (actually I think Phil Estes's manifest-tool does this) first makes those into untagged manifests in the lists's repo.

[jonjohnsonjr]

ICR does allow you to delete a manifest which is referenced from a manifest-list

Interesting. Does ICR require that all manifests referenced by a manifest-list exist in the repository when you upload the manifest-list?

Can you delete a blob that's referenced by a manifest?

Does ICR require that all blobs referenced by a manifest exist in the repository when you upload a manifest?

[brackendawson]

Interesting. Does ICR require that all manifests referenced by a manifest-list exist in the repository when you upload the manifest-list?

Yes. But it currently uses distribution's manifest statter to check this so that will also pass the check if a blob is linked in the repo.

Can you delete a blob that's referenced by a manifest?

Same as distribution, I could be wrong but I don't think you can delete a blob?
No, we don't currently authorize users for delete actions.

Does ICR require that all blobs referenced by a manifest exist in the repository when you upload a manifest?

Yes, same as distribution.

[jonjohnsonjr]

I could be wrong but I don't think you can delete a blob?

https://github.com/opencontainers/distribution-spec/blob/main/spec.md#deleting-blobs

[brackendawson]

Yea, I was just checking. We don't ever give users tokens with the delete action, so you can't delete blobs on ICR today.

[brackendawson]

Following up again, distribution allows blobs to be deleted using the * action and ICR will give users with the Writer role a token for the * action. This can be used to delete a blob.

Distribution does allow you to delete a blob that's referenced by the manifest and you can then no longer pull that image if you also need to pull that blob. The blob is not actually deleted, only the blob link in the repository, any other repos that still link the same blob will allow users to pull it. The blob will even still exist in storage after a delete if only a single repository ever linked it. ICR will behave differently in that the blob in storage can eventually be deleted if manifests referencing it are also deleted.

[rchincha]

zot Project [1]

• Conforms to dist-spec with regards to deletion
• Additionally, also supports garbage collection of blobs controlled via configuration [2]

References:
[1] https://github.com/anuvu/zot
[2] https://github.com/anuvu/zot/blob/master/examples/config-conformance.json#L5

[michaelb990]

Amazon ECR

• Official AWS documentation link.
• ECR provides 2 endpoints to interact with an ECR repository. An "AWS endpoint" that the AWS CLI & SDKs connect to, supports AWS SigV4 authentication, and is not OCI compatible. For OCI clients, we provide a second "OCI/Docker endpoint" that supports a subset of actions and implements the OCI Distribution spec.

Automatic Deletion

• ECR will not automatically delete manifests regardless of their tag status (tagged or untagged).
• ECR will automatically clean up layer blobs if they are not referenced by an image manifest.

Customer-driven Deletion

• ECR offers lifecycle policies that allow customers to define rules for deleting images that were pushed >N days ago or once >M images match a given tag status or tag prefix. Images that are referenced by an existing image index are protected from being deleted by an LCP rule. Customers are able to configure these rules per repository.
• ECR offers an API (ecr:BatchDeleteImage) for deleting images or tags via our AWS endpoint. We do not support deleting images via our OCI / Docker API endpoint.
• ECR will protect images that are referenced by an image index from being deleted via a call to ecr:BatchDeleteImage until the parent index is deleted.