support docker v2.2 manifest lists for multi-arch images

[ixdy]

Consider this a general brainstorming issue, since I'm not really sure how exactly this would work, and I think most bazel rules are missing necessary support for this anyway.

But as motivating example: the go_binary rule from bazelbuild/rules_go lets you set goos and goarch attributes, so you can theoretically create multiple copies of a go_binary with different architecture settings and build all of them in a single bazel build.

It then isn't a far stretch to imaging wrapping these into appropriate base images and then bundling them into a single manifest list to push to your docker registry of choice.

Existing (non-Bazel) work around manifest lists includes https://github.com/estesp/manifest-tool.

[mattmoor]

Having the capacity to produce fat images via Bazel is something I have long considered a goal of these rules, but for a long time Bazel has lacked the capacity for even basic cross compilation.

If I can simultaneously build two go_binary targets for distinct platforms, that's fantastic progress, but I'm not sure the Go rules alone supporting this would be enough.

I'd love a consistent story for producing these fat images across languages, so that we could write something like:

fat_image(
   name = "fat",
   image = "//path/to:image",  # This could be anything
   platforms = [
      ...
   ],
)

... and under the hood, we could synthesize a target that stitches together the image for each platform, the dependency on which is expressed by something like:

  "//path/to:image#--options"

I recall seeing something like that in a Bazel config doc once upon a time (re: generalized fat binary production), but have no idea where it went.

It is notable that the underlying library fully supports fat images, so the primary reason this doesn't exist is the lack of a good story for expressing it in Bazel.

@davidstanke FYI

@dlorenc @r2d4 FYI, since we'd probably want decent distroless image coverage (but not required)

[chrislovecnm]

👍

[ixdy]

per bazelbuild/bazel#5575 (comment), this might be implementable now.

[gregestren]

I'd be happy to chat feasibility with an appropriate rule owner expert.

[ixdy]

cc @nlopezgi @xingao267

[nlopezgi]

@ixdy thanks for creating this issue,
I'd be happy to talk with @gregestren about effort / feasibility of making this happen, but likely closer to end of this Q, beginning of next Q once we have cleared other pending issues from our backlog. If anyone wants to take the initiative to work on this before that we can coordinate to make it happen.

[fejta]

Any progress on this?

openshift/ci-tools#57

[nlopezgi]

None. I really haven't planned any time for this as I'm not sure who needs it and what impact building this feature would have. Happy to talk with folks interested in this to estimate effort / feasibility.

[Pensu]

@nlopezgi This would be helpful in creating multi-arch images for multiple projects that are using bazel for building like kubernetes/test-infra repo and the depending repos like openshift/ci-tools as well.

[nrvnrvn]

or for projects that are using bazel for building images and want to align their supported architectures with kubernetes and openshift.

[nlopezgi]

thanks @nrvnrvn and @Pensu,

Could you provide concrete examples of which projects are currently requiring this / would like this? I'm always interested in knowing which projects are using rules_docker (we love when folks send PRs adding their projects to the adopters list: https://github.com/bazelbuild/rules_docker/#adopters).

If we get a large / significant set of projects (that are in our adopters list) which are requesting this feature, it would greatly help with prioritization, and might just make it happen!

[Pensu]

@nlopezgi Prow is already there in the adopters list. :)

[munnerz]

👋 just dropping in to say the cert-manager project uses Bazel & rules_docker, and would love to get manifest list support so we can push multi-architecture docker images to our registry!

[jonjohnsonjr]

I'm interested in this for distroless. I'd like some minimal cross-platform base images for ko-build/ko#6

For my own purposes, I actually only care about gcr.io/distroless/static, for which I think this is actually trivial -- we'd just need to change each image's config file to have an appropriate os/arch, since it doesn't contain any binaries. Unfortunately, to do this generically for rules_docker is much more work 😭

[jonjohnsonjr]

(Just to track things.)

The k8s image promoter currently shells out to docker to work around this.

[rmohr]

https://github.com/kubevirt/kubevirt is also interested in this. kubevirt/kubevirt#2944 adds ppc64 support to our project and being able to build and push everything easily would be a great simplification.

[xin3liang]

Any progress on this issue/feature? @nlopezgi @gregestren
Strong need for k8s projects, because many of them are rebased on distroless base image. Without multi-arch support means current official k8s images( such as csi, cloud provider etc) can't run on other arch , such as arm64, ppc64le.
See:
https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/20190316-rebase-images-to-distroless.md
GoogleContainerTools/distroless#406

[nlopezgi]

No progress to report. Feature work for rules_docker is mostly depending on community contributions at this point. Happy to work with anyone that wants to take a stab at this one (not a trivial feature to implement though iiuc, but happy to work with anyone that wants to invest the time to make it happen).

[Pensu]

@nlopezgi I would love to help out with this. Let me know whatever is required to get this done. :)

[nlopezgi]

Thanks @Pensu ,
First step would be to start work on a design doc outlining the details of the feature to build and how it will be implemented. I haven't honestly looked in much detail at what is required to build this, but @jonjohnsonjr had some ideas iirc. If you want to start a shared doc and get some content started we can take the conversation there.

[jonpspri]

I just ran across this for some work I'm doing and was missing the same thing. I'm building container images entirely from scratch, so for the moment cross-compiling is not an issue. I'd envision something like:

multiarchitecture_manifest(
   name="...",
   fq_imagename="...", /* optional ? */
   images=[
      {  name="...", os="...", arch="..." },  /*  name refers to a RULE, either a container_pull or container_image  */
      ...
   ]
}

Ideally we'd drag some metadata out of the image builds, but I don't think that's quite hermetic. Anyway, it's a quick dump... for the moment I'm using a build control script outside of bazel to drive my fat image creation.

[jonjohnsonjr]

(dumping some constructive driveby feedback)

multiarchitecture_manifest

I'd probably call this container_image_index (or just image_index).

fq_imagename

We probably don't want to embed image name stuff in here. We can use a registry-independent on-disk representation like an image layout.

[jonpspri]

A rose by any other name... ? The main functional need I see is that whatever gets built needs to be pushable, and pushing implies pushing all underlying image areas.

Re fq_imagename: Hmm... right, we don't need image name if we build the manifest file ourselves. Shelling out to 'docker manifest ...' would be where image names come into play, which we want to avoid. I need to dig deeper into how bazel interacts with the Docker API and how the "simple" images are laid out on disk.

[hakman]

@nlopezgi Slightly related, I see that there is at least the possibility to create images with architectures other than AMD64 from #1427.
Would it be possible to have a new release containing this?
Thanks :)

[chrischdi]

@nlopezgi Slightly related, I see that there is at least the possibility to create images with architectures other than AMD64 from #1427.
Would it be possible to have a new release containing this?
Thanks :)

The current release already supports this and it is possible to build images for multiple platforms.
I just tested this using a go binary.

However, it is currently not possible to directly use the architecture argument in e.g. go_image because it is using app_layer under the hood, which currently does not allow to pass the architecture argument.

Would it be reasonable to add the architecture arg in app_layer and pass it to container_image (which does already support it)?

[jonjohnsonjr]

I'll drop another strawman in here:

container_image_index(
  name="whatever",
  manifests=[
    ":some_image",
    ":some_index",
    ":some_layer",
    ":some_descriptor",
  ],
  annotations={
    "foo": "bar",
  },
)

For :some_image, we can deduce the platform information by looking at the image configuration.

:some_index and :some_layer are self-explanatory, they would produce child descriptors that reference another index or a layer, which I believe we have enough info to generate, but if we don't...

:some_descriptor would allow us to override attributes of the manifest descriptor (or create arbitrary descriptors), something like:

container_descriptor(
  name="some_descriptor",
  media_type="application/vnd.oci.image.index.v1+json",
  size="1337",
  digest="sha256:d3adb33f...",
  platform={
    "architecture": "arm64",
    "os": "linux",
    "variant": "v8",
  },
)

If we're describing another rule, we can omit most of this, since bazel can figure it out, but maybe we need to override the platform because it's missing the variant or something:

container_descriptor(
  name="some_descriptor",
  image=":some_arm64_image",
  platform={
    "architecture": "arm64",
    "os": "linux",
    "variant": "v8",
  },
)

Last time I looked at this, bazel wasn't actually capable of doing cross-platform builds in one invocation of bazel. You'd need to do each cross-platform build separately, serialize them in some way, then have our index rule reference the serialized form to produce the manifest list as a final bazel invocation.

cc @imjasonh

[mattmoor]

Last time I looked at this, bazel wasn't actually capable of doing cross-platform builds in one invocation of bazel

Look at go_binary which support goarch 😉

[jonjohnsonjr]

Look at go_binary which support goarch

Isn't this specifically a go-ism that doesn't work for any other compiled language?

[jonjohnsonjr]

Last time I looked at this, bazel wasn't actually capable of doing cross-platform builds in one invocation of bazel. You'd need to do each cross-platform build separately, serialize them in some way, then have our index rule reference the serialized form to produce the manifest list as a final bazel invocation.

@katre can you speak to the accuracy of this? Is it possible to cross-compile for multiple platforms in one bazel invocation?

[mattmoor]

Isn't this specifically a go-ism that doesn't work for any other compiled language

Yup, but why would anyone use another language? 😉

(ducks)

[katre]

At this point, there's no easy way to have multiple target platforms in the same build. We intend to support that at some point, but there are numerous issues due to Bazel's memory usage that make it difficult. See bazelbuild/bazel#6519 for details and suggestions, but note that we aren't currently prioritizing the work until we can deal with the underlying performance issues.

[jonjohnsonjr]

I'm wondering, for prototyping this kind of thing, would it be reasonable to develop this under contrib/? Or should I just fork rules_docker? I'd like to be able to get folks to try this and give feedback before we commit to an interface, because I really doubt we'll get this right the first time around...

[mattmoor]

contrib/ seems like a good place to start with a pretty low commitment to compat.

[pcj]

Is anyone still interested in this? I think this may be out of scope for rules_docker itself given the lack of support noted by @katre and the complexity involved. This may be better suited to a separate project (rules_multidocker anyone?) that could be upstreamed if it worked out.

I'm going to close this but happy to re-open if someone is seriously considering working on it.

[jonjohnsonjr]

I am! I think it's still useful even if you can't build and push in one shot.

I'd like to replace container_bundle and the the on-disk tar format with an image layout (#1695), which would get this essentially for free.

Mostly, I just need to brainstorm with a bazel expert to help me map the OCI/docker abstractions onto bazel abstractions in an idiomatic way.

[kwiesmueller]

I'd also be interested as it would increase support for operators in a mixed arch kubernetes cluster.
Right now it's hard to roll out operators built with bazel on clusters that mix with arm64 for example.