Publish multi-arch image manifest

[rifelpet]

We're working on adding ARM64 support in Kops but in order to do that we need to use multi-architecture images for all of our addons because the nodes they could be scheduled on could span multiple architectures. Currently the CPA releases have separate images per architecture but it would be great to have a single image manifest for all architectures.

Adding support for this could be as simple as adding these docker manifest commands to the Makefile. Any objection to supporting this? We can provide assistance if needed.

[MrHohn]

/assign

[MrHohn]

kubernetes/k8s.io#1096 is promoting the 1.8.2 staging images to prod.

[MrHohn]

CPA 1.8.2 has been released: https://github.com/kubernetes-sigs/cluster-proportional-autoscaler/releases/tag/1.8.2

And a multi-arch image is available on k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.2

[hakman]

Thanks @MrHohn :)

[hakman]

@MrHohn I think something went wrong with the multi-arch image pushing. Could you check?

$ docker pull k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.2 --platform arm64
1.8.2: Pulling from cpa/cluster-proportional-autoscaler
no matching manifest for linux/arm64 in the manifest list entries

[johngmyers]

/reopen

[k8s-ci-robot]

@johngmyers: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MrHohn]

Yep looks like the multi-arch image is wrong:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 945,
         "digest": "sha256:e310b7e60ed5dabcf9945fb86ee049bafde69b01ac0b9eb647c2b96c273e18d8",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 945,
         "digest": "sha256:ba9d51870f9a718f91fb2a0df6366da36b2e5484a3331049b226310b5d59993d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 945,
         "digest": "sha256:549b40bf6c29920f0f023fd97a6a9197f785f15799523c3486456a1bd24c840f",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 945,
         "digest": "sha256:a9ba5e939a4f8f0ded4a6b222d01b6922fce231f7977098112d51e409fce81f0",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

[MrHohn]

Can't tell how the push command is wrong though (local run against my own gcr repo):

docker manifest create gcr.io/zihongz-k8s-dev/cluster-proportional-autoscaler:1.8.2 \
  --amend gcr.io/zihongz-k8s-dev/cluster-proportional-autoscaler-amd64:1.8.2 \
  --amend gcr.io/zihongz-k8s-dev/cluster-proportional-autoscaler-arm:1.8.2 \
  --amend gcr.io/zihongz-k8s-dev/cluster-proportional-autoscaler-arm64:1.8.2 \
  --amend gcr.io/zihongz-k8s-dev/cluster-proportional-autoscaler-ppc64le:1.8.2

@johngmyers Any idea?

[MrHohn]

Humm looks like cross-build image will be labeled with a wrong architecture.. moby/moby#35158

We can probably edit the architecture afterward via docker manifest annotate, for instance:
https://github.com/kubernetes/dns/blob/4e5fdbaa4659e2d4c28b880c9212077ba1cd313e/rules.mk#L78-L85

[johngmyers]

Yeah, I can't figure out a way to correct the arch metadata for the single-arch images. Filed #94.

[MrHohn]

kubernetes/k8s.io#1131 is promoting the new images that include the fix.

[MrHohn]

1.8.3 is out: https://github.com/kubernetes-sigs/cluster-proportional-autoscaler/releases/tag/1.8.3

Hope that fixes the issue!

[hakman]

Thanks for the help @MrHohn. It works perfectly:
https://testgrid.k8s.io/kops-misc#kops-aws-arm64-conformance