Build fails if containerd is used with an untrusted builder

[modulo11]

Summary

Using the containerd backend in Docker together with an untrusted builder fails the build:

ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:6528df63e9cafd1747f04fc57cad7b3910dce6be0e6cf49e8174fb2aeda69ef6" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:2fc1acc6fd5a97e82e2b780700a2c07e813e84fac1fe5857b42f475e6dc526d0: not found

Docker Desktop enabled the containerd backend for new installations starting with v4.34.0.

Reproduction

Steps

1. Enable containerd backend
2. git clone https://github.com/paketo-buildpacks/samples.git
3. cd samples/nodejs/npm
4. pack --verbose build --clear-cache --builder paketobuildpacks/builder-jammy-base:latest sample-node

Current behavior

Build fails.

Expected behavior

Build succeeds.

Environment

pack info

Pack:
  Version:  0.35.1+git-3a22a7f.build-6099
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.20.0

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13

Config:
(no config file found)

docker info

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 27.2.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.655GiB
 Name: docker-desktop
 ID: 1a97794d-938a-4f61-afee-1ce0744f88b0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal

[jjbustamante]

@modulo11

Have you tried this on an intel Macos machine? I think paketobuildpacks/builder-jammy-base builder is not available for ARM yet

[modulo11]

No, I sadly cannot test on Intel. Building works on my machine if I disable containerd, both use AMD64 builder images.

What I noticed is that imgutil uses different code paths when for saving if containerd is enabled or not. I was unsure where to file the issue, but I can open is there as well if you like.

[jjbustamante]

Yeah, if I remember correctly, using containerd has a performance penalty right now, some context here and we have that different path on imgutil, I will try to reproduce it on intel with containerd

[natalieparellano]

It's weird that trusted / untrusted seems to make a difference here. What image are we trying to fetch the base layers for?

[natalieparellano]

FWIW the error message seems to be coming from here: https://github.com/containerd/containerd/blob/61f91b963ef244daec1bda6700fe3f0b1aee50c6/core/images/archive/exporter.go#L335

I haven't found any references to this error in issues in any of buildpacks/moby/containerd. Would take some digging to understand how this could occur.

This reminds me of this issue we encountered when iterating between storage drivers. I know this is unsatisfying, but could you perhaps try clearing your image cache to see if that helps at all?

[natalieparellano]

@modulo11 any further thoughts on this one?

[modulo11]

Not really. After #2266 has been merged, the situation improved a bit for me.

[jjbustamante]

@modulo11

Based on your comment I will close this issue for now, if something comes up, feel free to reopen it and leave us a comment.

[remiphilippe]

FWIW, you can recreate this pretty consistently (not exactly the same message, and I'm running on Mac intel, in a devcontainer) with skaffold. There is probably an issue on skaffold side too, but if you're looking for reproductions, here it is.
pack.Build is called here: https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/build/buildpacks/lifecycle.go#L123-L139

Not Working:
Skaffold config:

build:
  artifacts:
    # Job Service - ListJobs Function
    - image: job_list
      context: go/functions
      buildpacks:
        builder: gcr.io/buildpacks/builder:google-22
        env:
          - GOOGLE_FUNCTION_SIGNATURE_TYPE=http
          - GOOGLE_FUNCTION_TARGET=ListJobs

error (I patched this skaffold version to run 0.36.0 of pack, to see if I could get a better error):

Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
google-22: Pulling from buildpacks/builder
Digest: sha256:afb313a7ffc6a63057ad8379d721b5c07a5d9b83ab54af411401db0bcd3fe6f9
Status: Image is up to date for gcr.io/buildpacks/builder:google-22
latest: Pulling from buildpacks/google-22/run
Digest: sha256:4cbbb4387c9c5eebbe5b24d0bb1cc03f8d701d7abdb8730c831a2b024cc2e229
Status: Image is up to date for gcr.io/buildpacks/google-22/run:latest
0.17.7: Pulling from buildpacksio/lifecycle
Digest: sha256:42e2ea1eec79c769a4c301cbb3536890d8d3860effde4034d8c6fc57f93f9587
Status: Image is up to date for buildpacksio/lifecycle:0.17.7
failed to fetch base layers: open /tmp/imgutil.local.image.3010387241/blobs/sha256/e68b7b3f7de4d6a37c32b138c19ddb2dd14c2ec7496fa186efc92eec3c309818: no such file or directory

With the "stock" lib of skaffold v2.13.2 (0.35.1):

google-22: Pulling from buildpacks/builder
Digest: sha256:afb313a7ffc6a63057ad8379d721b5c07a5d9b83ab54af411401db0bcd3fe6f9
Status: Image is up to date for gcr.io/buildpacks/builder:google-22
latest: Pulling from buildpacks/google-22/run
Digest: sha256:4cbbb4387c9c5eebbe5b24d0bb1cc03f8d701d7abdb8730c831a2b024cc2e229
Status: Image is up to date for gcr.io/buildpacks/google-22/run:latest
0.17.7: Pulling from buildpacksio/lifecycle
Digest: sha256:42e2ea1eec79c769a4c301cbb3536890d8d3860effde4034d8c6fc57f93f9587
Status: Image is up to date for buildpacksio/lifecycle:0.17.7
failed to write image to the following tags: [pack.local/builder/69646c776d7563736372:latest: saving image "pack.local/builder/69646c776d7563736372:latest": Error response from daemon: No such image: sha256:24288379d65c6bd95f9c826bdb12e364a23e365dc918727362a9df797e22bf53]```

Working
(for some reason I can only get it to work with pack 0.36.0)
Skaffold config:

build:
  artifacts:
    # Job Service - ListJobs Function
    - image: job_list
      context: go/functions
      buildpacks:
        trustBuilder: true
        builder: gcr.io/buildpacks/builder:google-22
        env:
          - GOOGLE_FUNCTION_SIGNATURE_TYPE=http
          - GOOGLE_FUNCTION_TARGET=ListJobs

Works fine:

Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
google-22: Pulling from buildpacks/builder
Digest: sha256:afb313a7ffc6a63057ad8379d721b5c07a5d9b83ab54af411401db0bcd3fe6f9
Status: Image is up to date for gcr.io/buildpacks/builder:google-22
latest: Pulling from buildpacks/google-22/run
Digest: sha256:4cbbb4387c9c5eebbe5b24d0bb1cc03f8d701d7abdb8730c831a2b024cc2e229
Status: Image is up to date for gcr.io/buildpacks/google-22/run:latest
PACK_VOLUME_KEY is unset; set this environment variable to a secret value to avoid creating a new volume cache on every build
===> ANALYZING
Timer: Analyzer started at 2024-12-07T03:21:49Z
Image with name "job_list:latest" not found
Timer: Analyzer ran for 104.589µs and ended at 2024-12-07T03:21:49Z