podman push to "dir:" has wrong mediaType

[vbatts]

Is this a BUG REPORT or FEATURE REQUEST?:

kind bug

Description
both as format docker v2s2 and as oci, the layers say they're +gzip but are just plain tar archives.

Steps to reproduce the issue:

1. podman push vbatts/slackware dir:xx

2. podman push --format oci vbatts/slackware dir:xxx

Describe the results you received:

[vbatts@poly] {master *} ~/xx$ jq . manifest.json
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1644,
    "digest": "sha256:f53237604b1d11c1a11c7868097bcb51ba8b77191199d3ec7f64600b47cb96a1"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 95180800,
      "digest": "sha256:c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f"
    }
  ]
}
[vbatts@poly] {master *} ~/xx$ file c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f
c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f: POSIX tar archive (GNU)

and

[vbatts@poly] {master *} ~/xxx$ jq . manifest.json 
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:e35d29b9ca3ffefd56aa85397497bc974614f4097c467d93051c3cb0730e29bd",
    "size": 671
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f",
      "size": 95180800
    }
  ]
}
[vbatts@poly] {master *} ~/xxx$ file c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f 
c3d15b6cd3a11ba35b3d1e390aee7de0d959084e8c37df1f3a51339e58838a7f: POSIX tar archive (GNU)

Describe the results you expected:

expected layer mediaType to be gzip (of that the object were itself gzipped)

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:       0.10.2-dev
Go Version:    go1.11.2
Git Commit:    "85dbfb33f4c20ca75c86abd687fe0f8a31ce080e"
Built:         Thu Nov  8 13:45:44 2018
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: Unknown
    path: /usr/local/libexec/crio/conmon
    version: 'conmon version , commit: 605136242787b6c7e1c7c8233b74a14c9097e510'
  Distribution:
    distribution: slackware
    version: "14.2"
  MemFree: 2444173312
  MemTotal: 8224837632
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      commit: ccb5efd37fb7c86364786e9137e22948751de7ed
      spec: 1.0.1-dev
  SwapFree: 8359243776
  SwapTotal: 8359243776
  arch: amd64
  cpus: 4
  hostname: poly.batts.lan
  kernel: 4.19.9
  os: linux
  uptime: 3h 31m 44.37s (Approximately 0.12 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 2
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /home/vbatts/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 5
  RunRoot: /var/run/user/1000

[rhatdan]

Valintin can you take a look at this.

[vrothberg]

Sure 👍

[vrothberg]

I am not sure how to tackle it. The behaviour comes from issue containers/skopeo#425 and has been changed by PR containers/image#370 which changed the compression for the directory transport with https://github.com/containers/image/pull/370/files#diff-e29498c4bcf6aaaa545bcf30a1030503R101 but I don't see why this was needed.

@mtrma, @runcom, do you know why the behaviour changed?

There's also the dest-compress switch in skopeo-copy which will force compression. The description claims that Compress tarball image layers when saving to directory using the 'dir' transport. (default is same compression type as source) although the code tells me that the compression type doesn't default to the one of the source.

[vrothberg]

@mtrmac ^

[mtrmac]

Can someone write down the precise actual/expected results? I am not quite clear what the reported problem is, or maybe @vrothberg is seeing an independent problem from the one reported;

e.g. is it that the layer is/is not compressed, or that the mediaType does/does not match? Or both?

---

It is a known missing functionality of c/image that it does not change MIME types at all on compression/decompression of layers. There isn’t, AFAIK, a fundamental reason why it couldn’t be done.

Do note, though, that https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md does not define an “uncompressed layer” type at all, so for schema2 there wasn’t much point in implementing the MIME type edits. (OTOH, since, OCI has introduced compressed/uncompressed variants, we probably should support these.)

This lack of MIME type changes would probably explain the why after pulling a compressed image from a remote registry (is that how the image was created?), creating a copy copy of the image from containers-storage: (which is always uncompressed) still uses the original compressed MIME types.

(IIRC, Buildah has independently implemented such a MIME type mapping, when creating the images from containers, and it uses the natural (but not defined by the spec) uncompressed MIME type string for schema2 uncompressed layers.)

---

has been changed by PR containers/image#370 which changed the compression for the directory transport with https://github.com/containers/image/pull/370/files#diff-e29498c4bcf6aaaa545bcf30a1030503R101 but I don't see why this was needed.

This didn’t change the behavior; it only changed the type of the return value from bool to an enum, but the new Compress/PreserveOriginal correspond exactly to the original true/false values; containers/image#370 has added the Decompress option for docker-archive: etc., but that AFAICS did not change dir: at all.

---

The description claims that Compress tarball image layers when saving to directory using the 'dir' transport. (default is same compression type as source) although the code tells me that the compression type doesn't default to the one of the source.

Where? AFAICS dirImageDestination.DesiredLayerCompression returns PreserveOriginal unless specifically instructed otherwise.

[vbatts]

@mtrmac
the reported problem is that the tar archives are not compressed, but the descriptor that references them says that they are compressed. So, one or the other side needs to be adjusted. Personally, I'd say to only update the mime-type/mediaType string to state the payload is not compressed.

While their v2.s2 doc does not have it, their docker/distribution code does have it and supports it for docker and OCI formats.

[mtrmac]

While their v2.s2 doc does not have it, their docker/distribution code does have it and supports it for docker and OCI formats.

Yes, there is a MediaTypeUncompressedLayer definition in docker/distribution, but not a single use of it within that repo.

docker/docker does use that constant, but only internally within the implementation of docker push, to cause the layers to be compressed; it never exists outside of the memory of the daemon.

I have now noticed, though, that containerd does recognize the uncompressed MIME type (known in there as MediaTypeDockerSchema2Layer) as a possible value; I suppose that’s good enough an argument for creating such manifests.

[vrothberg]

Where? AFAICS dirImageDestination.DesiredLayerCompression returns PreserveOriginal unless specifically instructed otherwise.

That's stated in skopeo copy --help but I could not see any code that actually alters the compression besides Skopeo's --dest-compress flag. Obviously, that's another issue than the reported one (but maybe I am misinterpreting it).

[vrothberg]

I have now noticed, though, that containerd does recognize the uncompressed MIME type (known in there as MediaTypeDockerSchema2Layer) as a possible value; I suppose that’s good enough an argument for creating such manifests.

(Just to check my understanding of the code) Does it imply we need to update/alter the manifest once the image is copied? Is there code in place for such tasks?

[mtrmac]

Where? AFAICS dirImageDestination.DesiredLayerCompression returns PreserveOriginal unless specifically instructed otherwise.

That's stated in skopeo copy --help but I could not see any code that actually alters the compression besides Skopeo's --dest-compress flag.

What do you mean by “alters the compression”, or maybe by “the compression type doesn't default to the one of the source”?

The dirImageDestination.DesiredLayerCompression returns PreserveOriginal by default, and c/image/copy/copier.copyBlobFromStream in that case always uses the original blob without compressing/decompressing it. So, AFAICS, copies to dir do “default to [compression type] of the source”.

[mtrmac]

Does it imply we need to update/alter the manifest once the image is copied? Is there code in place for such tasks?

Yes, we are already doing that when compressing images, so that the manifest of the destination refers to results of the compression which we have actually uploaded. In c/image/copy, see copyBlobFromStream for the code that actually modifies the blob and updates the input srcInfo when creating its return value, the end of copyLayers for staging the modifications, and ultimately copyUpdatedConfigAndManifest for creating the modified manifest. Somewhere along that path the MIME types need updating.

[vrothberg]

I have some spare cycles until DevConf and will look into it.

[rhatdan]

@vrothberg Any movement on this?

[vrothberg]

Not yet. There were too many other things on my table since DevConf but the queue is getting smaller.

[rhatdan]

@vrothberg Still no progress?

[vrothberg]

Unfortunately not. The work on the copy-detection mechanism exploded into rewriting parts of the storage backend and I am continuously blocked by cleaning up breaking builds/tests due to API changes across the stack which is really slowing me down.

[baude]

@vrothberg any ideas?

[vrothberg]

Unfortunately, no progress for a longer time since issues with higher priority are on my table. I removed me from the assignment to not block others from taking over.

[rhatdan]

@mtrmac @vrothberg Any movement on this? Should I assign an @QiWang19 ?

[mtrmac]

This is primarily WIP containers/image#563, some changes to buildah (almost certainly) and podman (maybe, less likely) would also be involved.

[vrothberg]

Yes, there are some leftovers to iron out and we want to wait for zstd PR to be merged, so I can pick that up and add code for the new mime type.

[vrothberg]

This issue will be fixed with merging #4165

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[mheon]

#4165 is merged, closing

[rocketraman]

Does anyone know what version of podman contains those changes? I'm building with podman 2.0.3 using BUILDAH_FORMAT=docker and I see that the built image has a layer which incorrectly reports the mediatype for one layer as application/vnd.docker.image.rootfs.diff.tar.gzip, when in fact the correct media type is application/vnd.docker.image.rootfs.diff.tar.

Furthermore, when doing:

podman push gcr.io/image-with-incorrect-layer-mediatype:foo gcr.io/image-with-incorrect-layer-mediatype:bar

podman apparently "corrects" the layer mediatype (everything else, including the layer hashes remains exactly the same) and the server weirdly ends up with the tags on different images, where the only difference is that mediatype value on two of the layers.

[QiWang19]

From the changelog, it's included from v1.6.2-rc1

[rocketraman]

From the changelog, it's included from v1.6.2-rc1

Thanks. Looks like I have a different issue then. Probably this one: #6496.