Use diffids - fix pulling from containers-storage:

[cgwalters]

Today we're using https://github.com/containers/containers-image-proxy-rs which is a Rust binding to https://github.com/containers/skopeo/blob/main/cmd/skopeo/proxy.go which is an IPC interface to containers/image.

In our usage of this, we fetch by blob right now. Operating system upgrades for example work fine fetching from a remote registry (e.g. quay.io/fedora/fedora-coreos:stable) - we stream and uncompress the blobs, writing to ostree.

However, it doesn't work to fetch by blob ID (compressed tar) from containers-storage because it only has the uncompressed version. We need to fetch via diffid.

We need containers/skopeo#1495 to fetch the config blob, then the pull code should use it.

[mtrmac]

Alternatively the proxy could expose LayerInfosForCopy, to be closer to what the c/image/copy code does and assumes.

(Admittedly LayerInfosForCopy is a wart that shouldn’t exist in the first place… between exposing that and hard-coding containers-storage: in clients, I find it hard to decide which one I dislike more :) )

[cgwalters]

Just noting that https://github.com/RishabhSaini is planning to look at this

[nalind]

I'd second the suggestion to expose and use LayerInfosForCopy(), and then add LayerInfos() to it as well.

A v2s1 image isn't going to have any DiffID values in the OCI-format config blob that is synthesized for it (observe: skopeo inspect --config docker://registry.k8s.io/pause:3.0), so reading the config and assuming that the DiffID values it contains can be used as suitable input values for GetBlob() is not always going to work.

LayerInfos() and LayerInfosForCopy() can supply the set of values that GetBlob() is prepared to be asked for.

[cgwalters]

• Add a proxy API to serialize the result of LayerInfosForCopy (if returns nothing, call GetLayerInfos)
• We can probably directly just call json.Marshal on this and have it work (but it'd be better to add json tags to the struct most likely)
• We can call GetBlob on these digests, filtering out the blobs (compressed) that we already have.
• We may need to store the uncompressed blob ID in the ostree commit that stores the blob to make this more efficient