Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS podman build fails with Dockerfile not in context #13561

Closed
ghost opened this issue Mar 18, 2022 · 4 comments
Closed

macOS podman build fails with Dockerfile not in context #13561

ghost opened this issue Mar 18, 2022 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote

Comments

@ghost
Copy link

ghost commented Mar 18, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On OSX (darwin) running podman build with the -f pointing to a file that's outside the context directory tree, the build fails with no such file or directory error. This issue also affects building from stdin (-f-) for the same reason (since stdin is written to a tmp file)

Steps to reproduce the issue:

mkdir podman-file-out-of-context
cd podman-file-out-of-context
printf 'FROM ubi8/ubi-minimal:latest\n\nRUN echo "test"\n' > Dockerfile
mkdir context_dir
podman build -f Dockerfile context_dir
# OR
cat Dockerfile | podman build -f- context_dir

Describe the results you received:

Error: stat /var/tmp/libpod_builder041882528/build/Users/cameron/dev/podman-file-out-of-context/Dockerfile: no such file or directory

Describe the results you expected:

The image to build

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 4.0.2

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.15.18-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 562688000
  memTotal: 4104019968
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 10h 46m 58.21s (Approximately 0.42 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319416
  BuiltTime: Thu Mar  3 08:56:56 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/amd64
  Version: 4.0.2


Package info (e.g. output of rpm -q podman or apt list podman):

N/A

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 18, 2022
@github-actions github-actions bot added the remote Problem is in podman-remote label Mar 18, 2022
@ghost
Copy link
Author

ghost commented Mar 18, 2022

I did a bit of digging myself and I believe the issue is related to the filepath.Walk through the sources when building the tar file due to the skip on root dir:

return nil // skip root dir

I tried out this code and it seems to make the build work now:

diff --git a/pkg/bindings/images/build.go b/pkg/bindings/images/build.go
index c508cb767..cfba1efe0 100644
--- a/pkg/bindings/images/build.go
+++ b/pkg/bindings/images/build.go
@@ -557,6 +557,57 @@ func nTar(excludes []string, sources ...string) (io.ReadCloser, error) {
 				return
 			}
 
+			// define add file link process to be shared by walk process and single file processor
+			addFileLink := func(info os.FileInfo, filename string) error {
+				di, isHardLink := checkHardLink(info)
+				hdr, err := tar.FileInfoHeader(info, "")
+				if err != nil {
+					return err
+				}
+				hdr.Uid, hdr.Gid = 0, 0
+				orig, ok := seen[di]
+				if ok {
+					hdr.Typeflag = tar.TypeLink
+					hdr.Linkname = orig
+					hdr.Size = 0
+					hdr.Name = filename
+					return tw.WriteHeader(hdr)
+				}
+				f, err := os.Open(filename)
+				if err != nil {
+					return err
+				}
+
+				hdr.Name = filename
+				if err := tw.WriteHeader(hdr); err != nil {
+					f.Close()
+					return err
+				}
+
+				_, err = io.Copy(tw, f)
+				f.Close()
+				if err == nil && isHardLink {
+					seen[di] = filename
+				}
+				return err
+			}
+
+			// if source is a regular file, skip to process it
+			if info, err := os.Stat(s); err == nil && info.Mode().IsRegular() {
+				err = func() error {
+					excluded, err := pm.Matches(s) // nolint:staticcheck
+					if err != nil {
+						return err
+					}
+					if excluded {
+						return nil
+					}
+					return addFileLink(info, s)
+				}()
+				merr = multierror.Append(merr, err)
+				continue
+			}
+
 			err = filepath.Walk(s, func(path string, info os.FileInfo, err error) error {
 				if err != nil {
 					return err
@@ -577,41 +628,10 @@ func nTar(excludes []string, sources ...string) (io.ReadCloser, error) {
 				}
 
 				if info.Mode().IsRegular() { // add file item
-					di, isHardLink := checkHardLink(info)
-					if err != nil {
-						return err
-					}
-
-					hdr, err := tar.FileInfoHeader(info, "")
+					err := addFileLink(info, name)
 					if err != nil {
 						return err
 					}
-					hdr.Uid, hdr.Gid = 0, 0
-					orig, ok := seen[di]
-					if ok {
-						hdr.Typeflag = tar.TypeLink
-						hdr.Linkname = orig
-						hdr.Size = 0
-						hdr.Name = name
-						return tw.WriteHeader(hdr)
-					}
-					f, err := os.Open(path)
-					if err != nil {
-						return err
-					}
-
-					hdr.Name = name
-					if err := tw.WriteHeader(hdr); err != nil {
-						f.Close()
-						return err
-					}
-
-					_, err = io.Copy(tw, f)
-					f.Close()
-					if err == nil && isHardLink {
-						seen[di] = name
-					}
-					return err
 				} else if info.Mode().IsDir() { // add folders
 					hdr, lerr := tar.FileInfoHeader(info, name)
 					if lerr != nil {

Not sure if there would be other implications with this change though with other systems.

@ghost
Copy link
Author

ghost commented Mar 18, 2022

before changes:

make podman-remote-darwin && echo "FROM ubi8/ubi-minimal:latest" | bin/darwin/podman build -f- ~/dev/podman-file-out-of-context/context_dir
CGO_ENABLED=0 \
                GOOS=darwin \
                GOARCH=amd64 \
                go build \
                -mod=vendor  \
                -o bin/darwin/podman-mac-helper \
                ./cmd/podman-mac-helper
/Library/Developer/CommandLineTools/usr/bin/make \
                CGO_ENABLED=0 \
                GOOS=darwin \
                GOARCH=amd64 \
                bin/darwin/podman
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build \
                -mod=vendor  \
                -ldflags '-X github.com/containers/podman/v4/libpod/define.gitCommit=8ebb94f2ced7fb8fa60ae0b53522cc6d9d6d0240 -X github.com/containers/podman/v4/libpod/define.buildInfo=1647642121 -X github.com/containers/podman/v4/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v4/libpod/config._etcDir=/usr/local/etc -X github.com/containers/common/pkg/config.additionalHelperBinariesDir= ' \
                -tags "remote exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper containers_image_openpgp" \
                -o bin/darwin/podman ./cmd/podman
Error: stat /var/tmp/libpod_builder531946892/build/private/var/folders/6d/npjyy5zs5x98y8fbpnjb_16m0000gn/T/build2533223464: no such file or directory

after changes:

make podman-remote-darwin && echo "FROM ubi8/ubi-minimal:latest" | bin/darwin/podman build -f- ~/dev/podman-file-out-of-context/context_dir
CGO_ENABLED=0 \
                GOOS=darwin \
                GOARCH=amd64 \
                go build \
                -mod=vendor  \
                -o bin/darwin/podman-mac-helper \
                ./cmd/podman-mac-helper
/Library/Developer/CommandLineTools/usr/bin/make \
                CGO_ENABLED=0 \
                GOOS=darwin \
                GOARCH=amd64 \
                bin/darwin/podman
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build \
                -mod=vendor  \
                -ldflags '-X github.com/containers/podman/v4/libpod/define.gitCommit=8ebb94f2ced7fb8fa60ae0b53522cc6d9d6d0240-dirty -X github.com/containers/podman/v4/libpod/define.buildInfo=1647642154 -X github.com/containers/podman/v4/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v4/libpod/config._etcDir=/usr/local/etc -X github.com/containers/common/pkg/config.additionalHelperBinariesDir= ' \
                -tags "remote exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper containers_image_openpgp" \
                -o bin/darwin/podman ./cmd/podman
STEP 1/1: FROM ubi8/ubi-minimal:latest
COMMIT
--> 281e62f51d4
281e62f51d47ff41e03e8f9a66fcd8b0a80a1f06445f5b4c3831c591c6d72cbe

@ghost ghost changed the title OS X podman build fails with Dockerfile not in context macOS podman build fails with Dockerfile not in context Mar 19, 2022
@Luap99
Copy link
Member

Luap99 commented Mar 21, 2022

Duplicate of #13293
@cdoern PTAL at the suggested change

@Luap99 Luap99 closed this as completed Mar 21, 2022
@cdoern
Copy link
Contributor

cdoern commented Mar 21, 2022

thanks for the suggestion @Cameron-Kurotori I have a similar solution on my end and will push it to #13531 soon

the issue is, we do want to skip the root dir sometimes but the way nTar is written we are skipping basically everything so I just made a quick patch to check for some particular cases.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote
Projects
None yet
Development

No branches or pull requests

2 participants