image exporter: return image descriptor in response

[crazy-max]

fixes #2558

$ ./hack/binaries
$ sudo ./bin/buildkitd &
$ sudo ./bin/buildctl build --progress plain --frontend gateway.v0 --opt source=docker/dockerfile --metadata-file metadata.json --local context=. --local dockerfile=. --output type=oci,dest=test.tar
$ cat metadata.json  | jq

{
  "containerimage.buildinfo": {
    "sources": [
      {
        "pin": "sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300",
        "ref": "docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300",
        "type": "docker-image"
      }
    ]
  },
  "containerimage.config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66",
  "containerimage.descriptor": {
    "annotations": {
      "config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66",
      "org.opencontainers.image.created": "2022-02-08T21:28:03Z"
    },
    "digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3",
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "size": 506
  },
  "containerimage.digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3"
}

@tonistiigi About config.digest as annotation should this be removed like?:

buildkit/exporter/oci/export.go

Line 214 in a640b47

delete(desc.Annotations, exptypes.ExporterConfigDigestKey)

Signed-off-by: CrazyMax [email protected]

[crazy-max]

Not sure if TestDiffOnlyMerge failed test is linked to the cni stack trace we have. Any idea @sipsma?

[sipsma]

Not sure if TestDiffOnlyMerge failed test is linked to the cni stack trace we have. Any idea @sipsma?

Talked with @tonistiigi about this in Slack a few days ago when he encountered a similar stack trace in the TestDiffSingleDeleteLayerOnMerge test. It's still hard to see how it could be directly related to merge/diff (it could just be that there are a ton of merge/diff tests that each do a lot of builds, but not sure) and also seems completely distinct from the CNI stack trace. I will take a closer look at it now, but shouldn't be a blocker for this PR or anything.

[tonistiigi]

RawMessage.UnmarshalJSON() doesn't actually parse any JSON https://cs.opensource.google/go/go/+/refs/tags/go1.17.6:src/encoding/json/stream.go;l=271 . This seems to still work but I'm not sure how 🤔 (or maybe just small inputs work) . But I think anyway it is safer to try parsing it to actual map interface. I think it is enough if we validate that it is a JSON object and don't handle liternals and arrays. That should minimize any possibility of randomly hitting this. Eg. "MTIK" is a valid base64 encoded JSON. (You can actually add it as a test case).

If the parsing is ok then use the data as RawMessage and discard the parsing result.

[tonistiigi]

it is a JSON object

Let's also make sure it is a non-zero length JSON object.

[tonistiigi]

^

[tonistiigi]

That's not what I meant. Just json.Unmarshal into a map[string]interface{}. If it decodes without an error and len(map) > 0 then add the bytes to out as RawMessage. Otherwise, we show the value as the original string.

[crazy-max]

Sure. For the strip json null props func I was thinking it could be useful for cases like {"foo":null,"bar":"baz"}.

[tonistiigi]

I'm not sure what case it is? If the daemon already sends JSON then we should not transform it. The question is only how do we detect what values should be shown as-is and what should be decoded.

[crazy-max]

Was just preventive but agree we should not change it.

[tonistiigi]

There are integration tests using buildctl as well https://github.com/moby/buildkit/blob/master/cmd/buildctl/build_test.go#L48 that allow you to also cover the json decoding feature without reimplementing it in a test.

[crazy-max]

I've already expanded the testBuildMetadataFile one for the descriptor: https://github.com/moby/buildkit/pull/2610/files#diff-487f6d73514b6a9c5895f8c77586ce38dc2b108fb3ed93bd0d63452667a7c3c3R157-R164. Is it not enough?

[tonistiigi]

No that is fine. I'm ok with just removing this test completely then. All cases seem to be covered also in that buildctl test.

[tonistiigi]

@crazy-max Also need to update https://github.com/moby/buildkit/blob/master/docs/build-repro.md#exporter-response-metadata

[crazy-max]

@tonistiigi Done in #2476 (README and build-repro.md): adb6b1d