Layer caching does not work with --squash-all --layers using Containerfile

[GrabbenD]

Issue Description

Problems:

• Problem 1) layer cache with --squash-all --layers never works if Containerfile doesn't use COPY --from instruction.

  Example: this will rebuild every time (cache starts working if FROM/COPY lines are uncommented below):
  Containerfile.base

  FROM archlinux AS rootfs
  
  RUN pacman --noconfirm -Syu htop
  
  # UNCOMMENT FROM/COPY TO FIX CACHE
  #FROM scratch
  #COPY --from=rootfs / /

  Command:
  $ time sh -c 'podman build -f Containerfile.base -t localhost/base --squash-all --layers'

• Problem 2) layer cache --squash-all --layers breaks for the 2nd image (which uses 1st image as base) if the 1st image is rebuilt even though the 1st image was built from cache.

  Example: this sequence of commands will trigger rebuild for 2nd image every time (I don't have any workaround):
  Containerfile.base

  FROM archlinux AS rootfs
  
  RUN pacman --noconfirm -Syu htop
  
  FROM scratch
  COPY --from=rootfs / /

  Containerfile.host

  FROM localhost/base AS rootfs
  
  RUN pacman --noconfirm -S nano
  
  FROM scratch
  COPY --from=rootfs / /

  Commands:
  time sh -c 'podman build -f Containerfile.base -t localhost/base --squash-all --layers && podman build -f Containerfile.host -t localhost/host --squash-all --layers'

Steps to reproduce the issue

• Please see above.

Describe the results you received

• Please see above.

Describe the results you expected

• Problem 1) Layer caching with --squash-all --layers shouldn't require COPY --from to work
• Problem 2) Rebuilding base image shouldn't trigger a rebuild in host image if contents are the same

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 98.82
    systemPercent: 0.71
    userPercent: 0.47
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2013
  hostname: ostree
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.5-2-cachyos-lto
  linkmode: dynamic
  logDriver: journald
  memFree: 2257625088
  memTotal: 33649405952
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: /usr/lib/podman/netavark is owned by netavark 1.7.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.9.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 1h 31m 38.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 35
    paused: 0
    running: 0
    stopped: 35
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 214374158336
  graphRootUsed: 122821722112
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 550
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1693343961
  BuiltTime: Tue Aug 29 21:19:21 2023
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178-dirty
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

N/A

Additional information

N/A

[flouthoc]

I'll check this.

[flouthoc]

@GrabbenD For case 1.

The last step needs to be executed before it can be squash, following change was done intentionally here: containers/buildah#4013

You can have a workaround by including a dummy last step and it will cache pacman line correctly.

FROM archlinux AS rootfs

RUN pacman --noconfirm -Syu htop
RUN echo do-nothing

For case 2.

Cache matching works on history since first line is changed and is using different base, it will never match the cache ( this is expected behavior ) if you wanna cache deps I'd suggest exploring RUN --mount=type=cache which is a feature built for this use-case itself.

I hope above answers your question. Please feel free to re-open or comment if something is unanswered.

[GrabbenD]

Seems like I'm still experiencing this issue

• Case 1 / Problem 1 (from original post)

  You can have a workaround by including a dummy last step and it will cache pacman line correctly.

  This doesn't work with multi-stage builds (which uses COPY --from=rootfs /mnt /). I've tried using the echo before FROM in the second stage and after COPY

• Case 2 / Problem 2 (from original post)

  I'd suggest exploring RUN --mount=type=cache

  Sorry, I should have mentioned that the use-case is for OSTree and the idea is to have the:

  1. First stage act as a BUILDER to generate a new rootfs into /mnt.
  2. Second stage is the RUNTIME, it uses scratch image and COPY --from=rootfs /mnt /

  (It's enough to use COPY --from=rootfs / / like I've done above to reproduce this)

Edit: My bad. --squash doesn't allow layer caching and the proposed workaround works as intended when using: --squash-all --layers