distroless/static multi-arch

[mattmoor]

The distroless/static image is essentially just data: cacerts, tzdata, /etc/passwd, /tmp. (I do see a single binary pulled in: tzconfig, but I think ideally we'd even strip this)

In ko, we (@jonjohnsonjr 👀 ) just landed support for multi-arch, which is based around the set of platforms available in the base image (when you pass --platform=all). I hear that Jib is looking into multi-arch as well.

By default, ko uses gcr.io/distroless/static:nonroot, which is just linux/amd64, but because the image itself is basically just data it actually works fine on arm64 (if forced). As a workaround for this, today I whipped up the script below to effectively "cast" the amd64 image to an arm64 image and produce a manifest list.

In principle, the way most of distroless builds, it should be possible to "properly" produce the image for each architecture from the packages on the appropriate architecture mirror, but I'm a little worried about the lack of multi-arch support in rules_docker being a major blocker.

I'm sort of wondering about shipping an initial band-aid like the trick below 🥺

package main

import (
	"log"
	"github.com/google/go-containerregistry/pkg/authn"
	"github.com/google/go-containerregistry/pkg/name"
	v1 "github.com/google/go-containerregistry/pkg/v1"
	"github.com/google/go-containerregistry/pkg/v1/empty"
	"github.com/google/go-containerregistry/pkg/v1/mutate"
	"github.com/google/go-containerregistry/pkg/v1/remote"
	"github.com/google/go-containerregistry/pkg/v1/types"
)
func main() {
	ref, err := name.ParseReference("gcr.io/distroless/static:nonroot")
	if err != nil {
		log.Fatalf("ParseReference() = %v", err)
	}
	tag, err := name.NewTag("docker.io/mattmoor/distroless-static:nonroot")
	if err != nil {
		log.Fatalf("NewTag() = %v", err)
	}
	ropt := []remote.Option{
		remote.WithAuthFromKeychain(authn.DefaultKeychain),
	}
	desc, err := remote.Get(ref, ropt...)
	if err != nil {
		log.Fatalf("Get() = %v", err)
	}
	switch desc.MediaType {
	case types.OCIImageIndex, types.DockerManifestList:
		log.Fatalf("ref: %v, is an index! %q", ref, desc.MediaType)
	}
	img, err := desc.Image()
	if err != nil {
		log.Fatalf("Image() = %v", err)
	}
	cf, err := img.ConfigFile()
	if err != nil {
		log.Fatalf("ConfigFile() = %v", err)
	}
	if cf.OS != "linux" {
		log.Fatalf("cf.OS = %s, wanted linux", cf.OS)
	}
	arm64cf := cf.DeepCopy()
	arm64cf.Architecture = "arm64"
	arm64img, err := mutate.ConfigFile(img, arm64cf)
	if err != nil {
		log.Fatalf("mutate.ConfigFile() = %v", err)
	}
	adds := []mutate.IndexAddendum{{
		Add: img,
		Descriptor: v1.Descriptor{
			MediaType: desc.MediaType,
			Platform: &v1.Platform{
				OS:           cf.OS,
				OSVersion:    cf.OSVersion,
				Architecture: cf.Architecture,
			},
		},
	}, {
		Add: arm64img,
		Descriptor: v1.Descriptor{
			MediaType: desc.MediaType,
			Platform: &v1.Platform{
				OS:           arm64cf.OS,
				OSVersion:    arm64cf.OSVersion,
				Architecture: arm64cf.Architecture,
			},
		},
	}}
	idx := mutate.IndexMediaType(mutate.AppendManifests(
		empty.Index,
		adds...,
	), types.OCIImageIndex)
	if err := remote.WriteIndex(tag, idx, ropt...); err != nil {
		log.Fatalf("WriteIndex() = %v", err)
	}
}

[mattmoor]

cc @Chanseok @loosebazooka

[mattmoor]

cc @imjasonh

[mattmoor]

A few other issues referencing multi-arch from the backlog:

• docker multiarch image - add ppc64le/s390x architectures to docker image #222
• ARM version? #377
• Can we publish the list of distroless supported architectures? #346

[mattmoor]

Based on #346 (comment) is sounds like a multi-arch static image would be useful for Kubernetes (cc @tallclair).

[loosebazooka]

Could we do this directly in rules_docker? @smukherj1 (sorry if wrong person) any idea here?

We would maybe only need docker manifest list publishing support (if we plan on this mechanism vs just explicitly tagged images)

There is an arm64 debian support -- https://www.debian.org/releases/stable/arm64/ (amr64v8)
It also looks like debs is deprecated? Not sure what's going on there, but distroless kinda depends on that doesn't it?

[mattmoor]

Could we do this directly in rules_docker

Yes, all that should be needed is support for publishing manifest lists, and then duplicating the appropriate bits for the different architectures. This wouldn't need to solve the whole problem of orchestrating multi-arch binary builds, since AFAIK distroless doesn't include any of its own binary material into the resulting images.

It would be an interesting experiment to try and produce just the arm64 flavor from the debs via a parallel set of rules.

If we wanted to do this without touching rules_docker, then it should then be relatively trivial to leverage github.com/google/go-containerregistry to write a multi-arch push rule that takes the :foo.tar for each platform as inputs to publish an image 🤔

[mattmoor]

Thanks to @dims here's what K8s needs:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 741,
         "digest": "sha256:73a2fe6120bcbc843ad076bbc754bb56280fbde02473d8d9805456f972a6fbbc",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 741,
         "digest": "sha256:bf251eda43ebdbb746996dc92a46ca8fc1aca27867d49a703d7e7b341f1484da",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 741,
         "digest": "sha256:e6130efd54aa3c29cb482bf2cecdd86b1ad4bb06f433b38db39de5e37943ba60",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 741,
         "digest": "sha256:f1163241faea5f5d255b2b5e1d861e1c4fe5af652bc593e2179bda25f5ccd335",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 741,
         "digest": "sha256:136b0968965e8dfdf14c7e7c112e874db5c933507960ae923c985656c0bcf4fd",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

From: us.gcr.io/k8s-artifacts-prod/kube-apiserver@sha256:264a3d5bc19e898d10440e6c6f0d923547ec731dff5f14a481af2a48dff4b5cf

[namrata-ibm]

@mattmoor Changes associated with this issue & PR are breaking CSI multiarch generation builds on s390x and ppc64le. Would you please help in extending the support for these platforms as well, similar to what is done for arm?
/cc @msau42

[mattmoor]

@namrata-ibm I'd be happy to review PRs, but the main problem we're likely to hit is lack of Bazel support to test those platforms.

If other architectures were previously using this base, then they should be able to safely return to using gcr.io/distroless/static:{latest,nonroot}-amd64, which is what used to be published as :{latest,nonroot} respectively.

cc @dims @justaugustus

[mattmoor]

FWIW, I'd wager the bulk of this can be done via changes to:

distroless/updateWorkspaceSnapshots.sh

Lines 32 to 76 in 4516713

	cat > checksums.bzl <<EOF
	# WARNING!!!
	# DO NOT MODIFY THIS FILE DIRECTLY.
	# TO GENERATE THIS RUN: ./updateWorkspaceSnapshots.sh
	ARCHITECTURES = ["amd64", "arm64"]
	VERSIONS = [
	("debian9", "stretch"),
	("debian10", "buster"),
	]
	DEBIAN_SNAPSHOT = "$DEBIAN_SNAPSHOT"
	DEBIAN_SECURITY_SNAPSHOT = "$DEBIAN_SECURITY_SNAPSHOT"
	SHA256s = {
	"amd64": {
	"debian9": {
	"main": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"backports": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch-backports/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"updates": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch-updates/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"security": "`curl -s https://snapshot.debian.org/archive/debian-security/$DEBIAN_SECURITY_SNAPSHOT/dists/stretch/updates/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	},
	"debian10": {
	"main": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/buster/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"updates": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/buster-updates/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"security": "`curl -s https://snapshot.debian.org/archive/debian-security/$DEBIAN_SECURITY_SNAPSHOT/dists/buster/updates/main/binary-amd64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	},
	},
	"arm64": {
	"debian9": {
	"main": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"backports": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch-backports/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"updates": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/stretch-updates/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"security": "`curl -s https://snapshot.debian.org/archive/debian-security/$DEBIAN_SECURITY_SNAPSHOT/dists/stretch/updates/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	},
	"debian10": {
	"main": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/buster/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"updates": "`curl -s https://snapshot.debian.org/archive/debian/$DEBIAN_SNAPSHOT/dists/buster-updates/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	"security": "`curl -s https://snapshot.debian.org/archive/debian-security/$DEBIAN_SECURITY_SNAPSHOT/dists/buster/updates/main/binary-arm64/Packages.gz 2>&1 | sha256sum | cut -d " " -f 1`",
	},
	},
	}
	EOF