Wildcard in .containerignore combined with containerfile outside of build context leads to an error #20259

albnnc · 2023-10-04T16:56:33Z

Issue Description

podman build fails if containerfile is outside of build context and .containerignore contains something like this:

*
!file-1
!file-2

With similar .containerignore:

Using stdin as input fails

➜  podman-test l
total 8
drwxr-xr-x   4 albnnc  staff   128B Oct  4 19:36 .
drwxr-xr-x  75 albnnc  staff   2.3K Oct  4 19:38 ..
-rw-r--r--   1 albnnc  staff    14B Oct  4 19:35 .containerignore
-rw-r--r--   1 albnnc  staff     0B Oct  4 19:36 readme.md
➜  podman-test cat .containerignore 
*
!readme.md

➜  podman-test echo "FROM centos:7" | podman --debug build -f - --tag podman-test .
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build -f - --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
DEBU[0000] Skipping excluded path: /private/var/folders/mq/slhk00gn5pbfkz_yydlvp3wm0000gn/T/build3155891543 
Error: stat /var/tmp/libpod_builder3963441430/build/private/var/folders/mq/slhk00gn5pbfkz_yydlvp3wm0000gn/T/build3155891543: no such file or directory

DEBU[0000] Shutting down engines

Using containerfile outside of build context fails

➜  podman-test l 
total 8
drwxr-xr-x   4 albnnc  staff   128B Oct  4 20:23 .
drwxr-xr-x  76 albnnc  staff   2.4K Oct  4 20:08 ..
-rw-r--r--   1 albnnc  staff    14B Oct  4 20:23 .containerignore
-rw-r--r--   1 albnnc  staff     0B Oct  4 19:36 readme.md
➜  podman-test cat ../podman-test.containerfile 
FROM centos:7

➜  podman-test podman --debug build -f ../podman-test.containerfile --tag podman-test .
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build -f ../podman-test.containerfile --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
DEBU[0000] Skipping excluded path: /Users/albnnc/Code/podman-test.containerfile 
Error: stat /var/tmp/libpod_builder1047373603/build/Users/albnnc/Code/podman-test.containerfile: no such file or directory

DEBU[0000] Shutting down engines

Using default containerfile works

➜  podman-test l
total 16
drwxr-xr-x   5 albnnc  staff   160B Oct  4 20:30 .
drwxr-xr-x  76 albnnc  staff   2.4K Oct  4 20:08 ..
-rw-r--r--   1 albnnc  staff    14B Oct  4 20:23 .containerignore
-rw-r--r--   1 albnnc  staff    15B Oct  4 20:30 Containerfile
-rw-r--r--   1 albnnc  staff     0B Oct  4 19:36 readme.md
➜  podman-test podman --debug build --tag podman-test .
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
STEP 1/1: FROM centos:7
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:7...
Getting image source signatures
Copying blob sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc
Copying config sha256:8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
COMMIT podman-test
--> 8652b9f0cb4c
Successfully tagged localhost/podman-test:latest
Successfully tagged quay.io/centos/centos:7
8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
DEBU[0023] Called build.PersistentPostRunE(podman --debug build --tag podman-test .) 
DEBU[0023] Shutting down engines

Using -f ./Containerfile works

➜  podman-test podman --debug build -f ./Containerfile --tag podman-test .
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build -f ./Containerfile --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
STEP 1/1: FROM centos:7
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:7...
Getting image source signatures
Copying blob sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc
Copying config sha256:8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
COMMIT podman-test
--> 8652b9f0cb4c
Successfully tagged localhost/podman-test:latest
Successfully tagged quay.io/centos/centos:7
8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
DEBU[0022] Called build.PersistentPostRunE(podman --debug build -f ./Containerfile --tag podman-test .) 
DEBU[0022] Shutting down engines

Using -f ./non-standard.containerfile works

➜  podman-test l
total 16
drwxr-xr-x   5 albnnc  staff   160B Oct  4 20:36 .
drwxr-xr-x  76 albnnc  staff   2.4K Oct  4 20:08 ..
-rw-r--r--   1 albnnc  staff    14B Oct  4 20:23 .containerignore
-rw-r--r--   1 albnnc  staff    15B Oct  4 20:30 non-standard.containerfile
-rw-r--r--   1 albnnc  staff     0B Oct  4 19:36 readme.md
➜  podman-test cat non-standard.containerfile 
FROM centos:7

➜  podman-test podman --debug build -f ./non-standard.containerfile --tag podman-test . 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build -f ./non-standard.containerfile --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
STEP 1/1: FROM centos:7
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:7...
Getting image source signatures
Copying blob sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc
Copying config sha256:8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
COMMIT podman-test
--> 8652b9f0cb4c
Successfully tagged localhost/podman-test:latest
Successfully tagged quay.io/centos/centos:7
8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
DEBU[0027] Called build.PersistentPostRunE(podman --debug build -f ./non-standard.containerfile --tag podman-test .) 
DEBU[0027] Shutting down engines

Without similar .containerignore:

Using containerfile outside of build context works

➜  podman-test l
total 8
drwxr-xr-x   4 albnnc  staff   128B Oct  4 20:38 .
drwxr-xr-x  76 albnnc  staff   2.4K Oct  4 20:08 ..
-rw-r--r--   1 albnnc  staff    13B Oct  4 20:38 .containerignore
-rw-r--r--   1 albnnc  staff     0B Oct  4 19:36 readme.md
➜  podman-test cat .containerignore 

!readme.md

➜  podman-test podman --debug build -f ../podman-test.containerfile --tag podman-test .
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called build.PersistentPreRunE(podman --debug build -f ../podman-test.containerfile --tag podman-test .) 
DEBU[0000] SSH Ident Key "/Users/albnnc/.ssh/podman-machine-default" SHA256:f09ir+0us7ELAWg7353MNJHmS27Q8fQ24Bivv//Pk0w ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.7.0/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.config/containers/auth.json 
DEBU[0000] Found an empty credential entry "https://index.docker.io/v1/" in "/Users/albnnc/.docker/config.json" (an unhandled credential helper marker?), moving on 
DEBU[0000] No credentials matching docker.io found in /Users/albnnc/.dockercfg 
DEBU[0000] No credentials for docker.io found           
DEBU[0000] DoRequest Method: POST URI: http://d/v4.7.0/libpod/build 
STEP 1/1: FROM centos:7
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:7...
Getting image source signatures
Copying blob sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc
Copying config sha256:8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
COMMIT podman-test
--> 8652b9f0cb4c
Successfully tagged localhost/podman-test:latest
Successfully tagged quay.io/centos/centos:7
8652b9f0cb4c0599575e5a003f5906876e10c1ceb2ab9fe1786712dac14a50cf
DEBU[0026] Called build.PersistentPostRunE(podman --debug build -f ../podman-test.containerfile --tag podman-test .) 
DEBU[0026] Shutting down engines

Steps to reproduce the issue

Use .containerignore with something like *, !file.
Run podman build -f with containerfile outside of build context.

Describe the results you received

podman build fails.

Describe the results you expected

podman build succeeds.

podman info output

Data

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 97.8
    systemPercent: 1.16
    userPercent: 1.04
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.4.15-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1201278976
  memTotal: 2048487424
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.aarch64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.aarch64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 12m 50.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 2530349056
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549246
  BuiltTime: Tue Sep 12 23:07:26 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

The host is a M1 Pro MacBook.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

The text was updated successfully, but these errors were encountered:

Luap99 · 2023-10-04T17:07:09Z

Version: 4.1.0

Please remove the machine and create anew one. The version in the machine is way to old. Not sure why this did not get updated.

Note sure if this fixes the bug but I remember changes around the remote build part handling so it could be. Also what is you client version as reported by podman version.

albnnc · 2023-10-04T17:44:21Z

I've updated the machine and the task description above. Seems like the issue is a little more common than I previously wrote. Hopefully, the description will be helpful.

albnnc · 2023-10-04T17:49:33Z

podman version output:

➜  podman-test podman version
Client:       Podman Engine
Version:      4.7.0
API Version:  4.7.0
Go Version:   go1.21.1
Git Commit:   073183fe1723d7bda826b574437891976a958c65
Built:        Wed Sep 27 18:35:55 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.20.7
Built:        Tue Sep 12 23:07:26 2023
OS/Arch:      linux/arm64

albnnc · 2023-10-04T21:57:04Z

As far as I can understand, external containerfiles are getting added to the build context here, but are excluded by pattern matcher later. Maybe one should add containerfile to the dontexcludes even if it's external because of possible wildcards in ignorefile.

Out of curiosity

Why does dontexcludes variable have to include strings like !Containerfile, !.containerignore, "!"+containerfile+".containerignore"? Are these "possible" files a must for build to work correctly? I thought that the only required file is main containerfile.

It also strange to me that we're forcing it to be included in context by adding "!"+containerfile and not the "!/"+containerfile. I mean, if the we have .containerignore

Containerfile

and file structure like

<context>
|- some_dir
|   |- ...
|   `- Containerfile (2)
|- ...
|- .containerignore
`- Containerfile (1)

containerfile (1) has to be included since we're building it, but containerfile (2) is not, because we're ignoring every of them in .containerignore. But the initial value of dontexcludes will likely lead to the inclusion of both containerfiles. Or not?

rhatdan · 2023-10-05T11:27:25Z

Sounds reasonable. Interested in opening a PR?

albnnc · 2023-10-06T10:48:59Z

@rhatdan, I added a PR. It includes a few new new lines which are, however, differ from what I described above. Adding "external" file to dontexcldues didn't work, because PatternMatcher doesn't seem to be designed to handle absolute paths at all.

albnnc added the kind/bug Categorizes issue or PR as related to a bug. label Oct 4, 2023

github-actions bot added the remote Problem is in podman-remote label Oct 4, 2023

albnnc changed the title ~~Wildcard in .containerignore combined with stdin as input leads to an error~~ Wildcard in .containerignore combined with containerfile outside of build context leads to an error Oct 4, 2023

albnnc mentioned this issue Oct 6, 2023

fix: don't ignore containerfiles outside of build context #20288

Merged

openshift-ci bot closed this as completed in #20288 Oct 9, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jan 8, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jan 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wildcard in .containerignore combined with containerfile outside of build context leads to an error #20259

Wildcard in .containerignore combined with containerfile outside of build context leads to an error #20259

albnnc commented Oct 4, 2023 •

edited

Loading

Luap99 commented Oct 4, 2023

albnnc commented Oct 4, 2023

albnnc commented Oct 4, 2023

albnnc commented Oct 4, 2023

rhatdan commented Oct 5, 2023

albnnc commented Oct 6, 2023

Wildcard in .containerignore combined with containerfile outside of build context leads to an error #20259

Wildcard in .containerignore combined with containerfile outside of build context leads to an error #20259

Comments

albnnc commented Oct 4, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Luap99 commented Oct 4, 2023

albnnc commented Oct 4, 2023

albnnc commented Oct 4, 2023

albnnc commented Oct 4, 2023

rhatdan commented Oct 5, 2023

albnnc commented Oct 6, 2023

albnnc commented Oct 4, 2023 •

edited

Loading