Ephemeral builder image not deleted at the end of the build

[edmorley]

Summary

When a pack build is performed that creates an ephemeral builder image, that builder image is now left behind at the end of the build when it was previously deleted. After lots of builds in a row this results in many stray images left behind.

---

Reproduction

Steps

1. docker system prune --all --volumes --force
2. docker images
3. mkdir testcase && cd $_ && touch Procfile
4. pack build testcase --builder heroku/builder:24 --buildpack heroku/[email protected]
5. docker images | rg builder

Current behavior

Unexpected leftover ephemeral builder image:

$ docker images | rg builder
pack.local/builder/796e7271616672796575   latest    92e6cc20ac67   44 years ago   1.05GB
...

Expected behavior

No leftover ephemeral builder image.

---

Environment

pack info

• Pack CLI v0.35.1+git-3a22a7f.build-6099
• lifecycle v0.20.1

docker info

docker info

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/emorley/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/emorley/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/emorley/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/emorley/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/emorley/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/emorley/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/emorley/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/emorley/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: REDACTED
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/emorley/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

[jjbustamante]

We changed these lines with this commit maybe it is related?

cc @natalieparellano

[natalieparellano]

Thanks @jjbustamante. I would have to dig a little bit to determine if it's related to that commit. The ephemeral builder name should never equal the raw builder name if we actually created an ephemeral builder image. That said the timing of the bug does seem suspicious.

Does anyone have bandwidth to look into this?

[natalieparellano]

Oh jeez, I realized that createEphemeralBuilder actually mutates the provided the rawBuilderImage that is passed in, so ephemeralBuilder.Name() == rawBuilderImage.Name() always returns true. I think #2260 should fix it, but I haven't tested it.