container rules put high load on remote cache

[alexeagle]

I'm following up from a severe outage caused by a remote cache overload.

rules_docker tries to be careful to use layer tars as the primary outputs of rules, which is great. However there are lots of cases where an entire OCI image is the output of a rule, and this causes a ton of load on remote cache. In particular, we have some actions that run something fast (like gzip) to transform some large files into some other large files. These action results are not appropriate to send to the remote cache because under most conditions it's faster to perform the action again locally than to fetch from the remote cache.

As an illustration, take the example from the top of rules_docker README

WORKSPACE

load("@io_bazel_rules_docker//container:container.bzl", "container_pull")

container_pull(
  name = "java_base",
  registry = "gcr.io",
  repository = "distroless/java",
)

Then I just build that image (no BUILD files required for this repro!)

By instrumenting the remote cache to reject uploads over 10MB (see buchgr/bazel-remote#440) I can see that we attempt to upload three such blobs:

% bazel build @java_base//image --remote_cache=grpc://localhost:9092 --remote_retries=1 --execution_log_json_file=/tmp/execution_log.json
...
WARNING: Writing to Remote Cache:
Error while uploading artifact with digest '5d09c2db1d761a6cd292a453815df5fecfa98058e4f2c0185976d1f28dfc09ec/17786880'
WARNING: Writing to Remote Cache:
Error while uploading artifact with digest 'b04be07e97435bfe3590f9135809b12de5573155d9814e9c0bbaa3b56d7234ac/170127360'
WARNING: Writing to Remote Cache:
Error while uploading artifact with digest '1a3910ac3ab501fb4ed8210223ff77c5529b2d23aa737ce47161a812a2b185ad/201798144'
Target @java_base//image:image up-to-date:
  bazel-bin/external/java_base/image/image.tar

That's a 17MB object, a 162MB object, and a 192MB object being stored in the cache for a build that depends on the java_base image.

I think perhaps we should audit some of the actions that are expected to produce whole-image outputs, and mark these with tags=['no-remote-cache']

[alexeagle]

Here's a slightly reduced execution log showing the three spawns that resulted in the remote-cachable outputs above

{
  "commandArgs": [
    "bazel-out/host/bin/external/io_bazel_rules_docker/container/go/cmd/zipper/zipper_/zipper",
    "-src",
    "external/java_base/image/001.tar.gz",
    "-dst",
    "bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.nogz",
    "-decompress"
  ],
  "listedOutputs": [
    "bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.nogz"
  ],
  "remotable": true,
  "cacheable": true,
  "progressMessage": "GUNZIP external/java_base/image/001.tar.gz.nogz",
  "mnemonic": "GUNZIP",
  "actualOutputs": [
    {
      "path": "bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.nogz",
      "digest": {
        "hash": "5d09c2db1d761a6cd292a453815df5fecfa98058e4f2c0185976d1f28dfc09ec",
        "sizeBytes": "17786880"
      }
    }
  ],
  "runner": "linux-sandbox",
  "remoteCacheHit": false,
  "status": "",
  "exitCode": 0
}
{
  "commandArgs": [
    "bazel-out/host/bin/external/io_bazel_rules_docker/container/go/cmd/zipper/zipper_/zipper",
    "-src",
    "external/java_base/image/004.tar.gz",
    "-dst",
    "bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.nogz",
    "-decompress"
  ],
  "listedOutputs": [
    "bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.nogz"
  ],
  "remotable": true,
  "cacheable": true,
  "progressMessage": "GUNZIP external/java_base/image/004.tar.gz.nogz",
  "mnemonic": "GUNZIP",
  "actualOutputs": [
    {
      "path": "bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.nogz",
      "digest": {
        "hash": "b04be07e97435bfe3590f9135809b12de5573155d9814e9c0bbaa3b56d7234ac",
        "sizeBytes": "170127360"
      }
    }
  ],
  "runner": "linux-sandbox",
  "remoteCacheHit": false,
  "status": "",
  "exitCode": 0
}
{
  "commandArgs": [
    "bazel-out/host/bin/external/io_bazel_rules_docker/container/go/cmd/join_layers/join_layers_/join_layers",
    "--output=bazel-out/k8-fastbuild/bin/external/java_base/image/image.tar",
    "--experimental-tarball-format=legacy",
    "--tag=bazel/image:image=external/java_base/image/config.json",
    "--layer=external/java_base/image/000.tar.gz,bazel-out/k8-fastbuild/bin/external/java_base/image/000.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/java_base/image/000.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/java_base/image/000.tar.gz.nogz.sha256",
    "--layer=external/java_base/image/001.tar.gz,bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/java_base/image/001.tar.gz.nogz.sha256",
    "--layer=external/java_base/image/002.tar.gz,bazel-out/k8-fastbuild/bin/external/java_base/image/002.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/java_base/image/002.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/java_base/image/002.tar.gz.nogz.sha256",
    "--layer=external/java_base/image/003.tar.gz,bazel-out/k8-fastbuild/bin/external/java_base/image/003.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/java_base/image/003.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/java_base/image/003.tar.gz.nogz.sha256",
    "--layer=external/java_base/image/004.tar.gz,bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/java_base/image/004.tar.gz.nogz.sha256"
  ],
  "listedOutputs": [
    "bazel-out/k8-fastbuild/bin/external/java_base/image/image.tar"
  ],
  "remotable": true,
  "cacheable": true,
  "progressMessage": "JoinLayers external/java_base/image/image.tar",
  "mnemonic": "JoinLayers",
  "actualOutputs": [
    {
      "path": "bazel-out/k8-fastbuild/bin/external/java_base/image/image.tar",
      "digest": {
        "hash": "1a3910ac3ab501fb4ed8210223ff77c5529b2d23aa737ce47161a812a2b185ad",
        "sizeBytes": "201798144"
      }
    }
  ],
  "runner": "linux-sandbox",
  "remoteCacheHit": false,
  "status": "",
  "exitCode": 0
}