Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnf transaction fails inside Fedora container and corrupts container #108

Closed
yann-soubeyrand opened this issue Aug 21, 2019 · 33 comments · Fixed by #109
Closed

dnf transaction fails inside Fedora container and corrupts container #108

yann-soubeyrand opened this issue Aug 21, 2019 · 33 comments · Fixed by #109
Assignees

Comments

@yann-soubeyrand
Copy link

/kind bug

Description

Doing a dnf install inside a Fedora container or a Silverblue toolbox results in failed transaction and corrupted container.

Steps to reproduce the issue:

  1. podman --log-level=debug run -ti --restart=no --rm fedora

  2. Inside container: dnf install jq (jq is an example, it could be dnf upgrade)

Describe the results you received:

DNF transaction failed. If the command used is dnf upgrade the system is corrupted (library files from the packages in the transaction are deleted).

INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /var/home/yann/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/yann/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /var/home/yann/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/yann/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/fedora:latest" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] Got mounts: []                               
DEBU[0000] Got volumes: []                              
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] parsed reference into "[overlay@/var/home/yann/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] exporting opaque data as blob "sha256:ef49352c9c21ca58ce753fd0b5dda645492236cec4213ac31ee47c35f2e91b1c" 
DEBU[0000] created container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has work directory "/var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has run directory "/tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata" 
DEBU[0000] New container created "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" has CgroupParent "/libpod_parent/libpod-325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] overlay: mount_data=lowerdir=/var/home/yann/.local/share/containers/storage/overlay/l/A3V3TYSMAI2J2KG2KDT3Q66D2D,upperdir=/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/diff,workdir=/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/work,context="system_u:object_r:container_file_t:s0:c189,c381" 
DEBU[0000] mounted container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" at "/var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/merged" 
DEBU[0000] Created root filesystem for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 at /var/home/yann/.local/share/containers/storage/overlay/7b6774f1279f0318dcf5541cfd248594503b2f100c029d6b20e020432aded85a/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Created OCI spec for container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 at /var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[-c 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 -u 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 -n jovial_jang -r /usr/bin/runc -b /var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata -p /tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/pidfile --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /tmp/1000/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/yann/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 --socket-dir-path /run/user/1000/libpod/tmp/socket -t -l k8s-file:/var/home/yann/.local/share/containers/storage/overlay-containers/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/userdata/ctr.log --log-level debug --syslog]"
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied 
DEBU[0000] Received container pid: 19072                
DEBU[0000] Created container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 in OCI runtime 
DEBU[0000] Attaching to container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254/attach 
DEBU[0000] Received a resize event: {Width:238 Height:55} 
DEBU[0000] Starting container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 with command [/bin/bash] 
DEBU[0000] Started container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0000] Enabling signal proxying                     
[root@325b9572ad86 /]# dnf install jq
Fedora Modular 30 - x86_64                                                                                                                                                                                    1.1 MB/s | 2.7 MB     00:02    
Fedora Modular 30 - x86_64 - Updates                                                                                                                                                                          959 kB/s | 3.0 MB     00:03    
Fedora 30 - x86_64 - Updates                                                                                                                                                                                  5.0 MB/s |  19 MB     00:03    
Fedora 30 - x86_64                                                                                                                                                                                             12 MB/s |  70 MB     00:05    
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                   Architecture                                           Version                                                       Repository                                               Size
==============================================================================================================================================================================================================================================
Installing:
 jq                                                        x86_64                                                 1.6-2.fc30                                                    fedora                                                  168 k
Installing dependencies:
 oniguruma                                                 x86_64                                                 6.9.2-2.fc30                                                  updates                                                 193 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  2 Packages

Total download size: 361 k
Installed size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.9.2-2.fc30.x86_64.rpm                                                                                                                                                                      665 kB/s | 193 kB     00:00    
(2/2): jq-1.6-2.fc30.x86_64.rpm                                                                                                                                                                               462 kB/s | 168 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         193 kB/s | 361 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Installing       : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                        1/2 
Error unpacking rpm package oniguruma-6.9.2-2.fc30.x86_64
  Installing       : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                 2/2 
error: unpacking of archive failed on file /usr/lib/.build-id/1c/7588d6da78dd5888d79f988ab594f6b5abeeb5;5d5d2bf0: cpio: utime
error: oniguruma-6.9.2-2.fc30.x86_64: install failed

Error unpacking rpm package jq-1.6-2.fc30.x86_64
  Verifying        : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                        1/2 
  Verifying        : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                 2/2 

Failed:
  oniguruma-6.9.2-2.fc30.x86_64                                                                                              jq-1.6-2.fc30.x86_64                                                                                             

Error: Transaction failed
[root@325b9572ad86 /]# exit
DEBU[0471] Checking container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 status... 
DEBU[0471] Cleaning up container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0471] Network is already cleaned up, skipping...   
DEBU[0471] unmounted container "325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254" 
DEBU[0471] Successfully cleaned up container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 
DEBU[0471] Container 325b9572ad8632aff667e03952e69c7bba5a1d2cb7538f7f35ad88961e5c1254 storage is already unmounted, skipping...

Describe the results you expected:

I expect a successful installation of jq (or upgrade of the system if using dnf upgrade).

Additional information you deem important (e.g. issue happens only occasionally):

Issue did not happen a month ago. I don't know exactly when the problem appeared though.

Output of podman version:

Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 10731966464
  MemTotal: 16551071744
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 8342466560
  SwapTotal: 8342466560
  arch: amd64
  cpus: 8
  hostname: work-laptop
  kernel: 5.2.9-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 58m 49.5s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/yann/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/yann/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /var/home/yann/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.4.4-4.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora Silverblue 30.20190821.0.

@yann-soubeyrand
Copy link
Author

I forgot to mention that using Docker instead of Podman works as expected.

@rhatdan
Copy link
Member

rhatdan commented Aug 21, 2019

@giuseppe is this one of the bugs you fixed in fuse-overlay?

@rhatdan
Copy link
Member

rhatdan commented Aug 21, 2019

I would bet it works well with root podman as well.

@giuseppe
Copy link
Member

I think this is already fixed, but the rpm didn't hit Stable yet. Could you try with https://bodhi.fedoraproject.org/updates/FEDORA-2019-ed81918b28 ?

@giuseppe giuseppe transferred this issue from containers/podman Aug 22, 2019
@yann-soubeyrand
Copy link
Author

@giuseppe how can I test this package safely on Silverblue ?

@rhatdan
Copy link
Member

rhatdan commented Aug 22, 2019

@yann-soubeyrand I think you can download it and install it as a layered package using rpm-ostree.
I just tried it on Fedora 31 and the package installed correctly with the update fuse-overlayfs.

@yann-soubeyrand
Copy link
Author

@rhatdan it won't mess with my underlying ostree when I'll remove the layered package?

@rhatdan
Copy link
Member

rhatdan commented Aug 22, 2019

Nope, that is the way rpm-ostree layering works. They can update independently.
@cgwalters could explain further.

@cgwalters
Copy link

This wouldn't be a layer but an override.
Run:

rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/fuse-overlayfs/0.5.2/3.git4dc60f0.fc30/x86_64/fuse-overlayfs-0.5.2-3.git4dc60f0.fc30.x86_64.rpm

@rhatdan
Copy link
Member

rhatdan commented Aug 22, 2019

Sorry wrong terminology. Thanks @cgwalters

@yann-soubeyrand
Copy link
Author

It doesn't work either with fuse-overlayfs 0.5.2-3.git4dc60f0.fc30.

@yann-soubeyrand
Copy link
Author

But indeed, it works with root podman.

@rhatdan
Copy link
Member

rhatdan commented Aug 22, 2019

Well maybe you are seeing something different, we don't see this issue with f30.
Are you sure you started a new container and did not just exec into an already running container under toolbox. I don't think the fuse_overlay would be changed in a running container.

Perhaps reboot after replacying the fuse-overlayfs, actually I think that is required on silverblue anyways.

@giuseppe
Copy link
Member

I think I know what is going on, it is an issue exposed by recently enabling FUSE writeback. Going to take a look at it

@yann-soubeyrand
Copy link
Author

@rhatdan

Are you sure you started a new container and did not just exec into an already running container under toolbox. I don't think the fuse_overlay would be changed in a running container.

I don't have toolbox any more as dnf upgrade crashed it. I did a podman run so I was in a brand new container ;-)

Perhaps reboot after replacying the fuse-overlayfs, actually I think that is required on silverblue anyways.

It's indeed mandatory to take the new ostree into account and I did it ;-)

@giuseppe Cool! Working on Silverblue without toolbox is a bit incapacitating :-D

giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Aug 22, 2019
introduce a hash map to refer from an inode to the file paths.
A recent change where we enable FUSE writeback by default uncovered an
underlying issue in fuse-overlayfs where changes to a file with
multiple links would not be visible from the other link.

For each inode, maintain a list of nodes that refer to it, so that we
can still access it when a link is removed and more importantly we can
use the inode value with FUSE.

Closes: containers#108
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1744109

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Aug 23, 2019
introduce a hash map to refer from an inode to the file paths.
A recent change where we enable FUSE writeback by default uncovered an
underlying issue in fuse-overlayfs where changes to a file with
multiple links would not be visible from the other link.

For each inode, maintain a list of nodes that refer to it, so that we
can still access it when a link is removed and more importantly we can
use the inode value with FUSE.

Closes: containers#108
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1744109

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Aug 23, 2019
introduce a hash map to refer from an inode to the file paths.
A recent change where we enable FUSE writeback by default uncovered an
underlying issue in fuse-overlayfs where changes to a file with
multiple links would not be visible from the other link.

For each inode, maintain a list of nodes that refer to it, so that we
can still access it when a link is removed and more importantly we can
use the inode value with FUSE.

Closes: containers#108
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1744109

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/fuse-overlayfs that referenced this issue Aug 23, 2019
introduce a hash map to refer from an inode to the file paths.
A recent change where we enable FUSE writeback by default uncovered an
underlying issue in fuse-overlayfs where changes to a file with
multiple links would not be visible from the other link.

For each inode, maintain a list of nodes that refer to it, so that we
can still access it when a link is removed and more importantly we can
use the inode value with FUSE.

Closes: containers#108
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1744109

Signed-off-by: Giuseppe Scrivano <[email protected]>
@mvala
Copy link

mvala commented Aug 27, 2019

It is not fixed for me

[mvala@localhost ~]$ rpm -q fuse-overlayfs
fuse-overlayfs-0.6-2.git43b641d.fc30.x86_64
[mvala@localhost ~]$ podman run -ti --restart=no --rm fedora
[root@8810c24040b1 /]# dnf install jq -y
Fedora Modular 30 - x86_64                                                                                                                                                                                     1.3 MB/s | 1.9 MB     00:01    
Fedora Modular 30 - x86_64 - Updates                                                                                                                                                                           1.5 MB/s | 2.7 MB     00:01    
Fedora 30 - x86_64 - Updates                                                                                                                                                                                   5.2 MB/s |  23 MB     00:04    
Fedora 30 - x86_64                                                                                                                                                                                             4.8 MB/s |  61 MB     00:12    
Last metadata expiration check: 0:00:01 ago on Tue Aug 27 05:48:55 2019.
Dependencies resolved.
===============================================================================================================================================================================================================================================
 Package                                                   Architecture                                           Version                                                        Repository                                               Size
===============================================================================================================================================================================================================================================
Installing:
 jq                                                        x86_64                                                 1.6-2.fc30                                                     fedora                                                  168 k
Installing dependencies:
 oniguruma                                                 x86_64                                                 6.9.2-2.fc30                                                   updates                                                 193 k

Transaction Summary
===============================================================================================================================================================================================================================================
Install  2 Packages

Total download size: 361 k
Installed size: 1.2 M
Downloading Packages:
(1/2): jq-1.6-2.fc30.x86_64.rpm                                                                                                                                                                                667 kB/s | 168 kB     00:00    
(2/2): oniguruma-6.9.2-2.fc30.x86_64.rpm                                                                                                                                                                       131 kB/s | 193 kB     00:01    
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                          117 kB/s | 361 kB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                       1/1 
  Installing       : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                         1/2 
Error unpacking rpm package oniguruma-6.9.2-2.fc30.x86_64
  Installing       : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                  2/2 
error: unpacking of archive failed on file /usr/lib/.build-id/1c/7588d6da78dd5888d79f988ab594f6b5abeeb5;5d64c46a: cpio: utime
error: oniguruma-6.9.2-2.fc30.x86_64: install failed

Error unpacking rpm package jq-1.6-2.fc30.x86_64
  Verifying        : oniguruma-6.9.2-2.fc30.x86_64                                                                                                                                                                                         1/2 
  Verifying        : jq-1.6-2.fc30.x86_64                                                                                                                                                                                                  2/2 

Failed:
  oniguruma-6.9.2-2.fc30.x86_64                                                                                              jq-1.6-2.fc30.x86_64                                                                                             

Error: Transaction failed

@mvala
Copy link

mvala commented Aug 27, 2019

Same with

[mvala@localhost ~]$ rpm -q fuse-overlayfs
fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64

@giuseppe
Copy link
Member

It could be SELinux blocking it. Could you try again with SELinux disabled? We have fixed recently an issue like that, so you might need to update it

@mvala
Copy link

mvala commented Aug 27, 2019

Yes, with SELinux disabled it works. Will there be version with SELinux enabled?

@rhatdan
Copy link
Member

rhatdan commented Aug 27, 2019

container-selinux-2.115.0-1.gitfddfbbb.fc30 is the latest available.

@mvala
Copy link

mvala commented Aug 27, 2019

Thanks, i still have container-selinux-2.113.0-1.dev.git4f7d6bb.fc30.noarch

@rhatdan
Copy link
Member

rhatdan commented Aug 27, 2019

114 should be in updates testing now, and might have the fix.

dnf -y update --enablerepo=updates-testing container-selinux

@mvala
Copy link

mvala commented Aug 27, 2019

I use silverblue and i don't know yet how to enable testing repo in rpm-ostree

@cgwalters
Copy link

rpm-ostree rebase fedora/30/x86_64/testing/silverblue should work

@yann-soubeyrand
Copy link
Author

Hi,

114 seems not to fix the bug: having container-selinux-2:2.114.0-1.git028ab00.fc30.noarch and fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64 on Silverblue 30 and still hitting the bug.

@yann-soubeyrand
Copy link
Author

Hi!

I went through the following steps.

  • I rebased on testing:
[yann@work-laptop ~]$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree://fedora:fedora/30/x86_64/testing/silverblue
                   Version: 30.20190908.0 (2019-09-08T03:30:48Z)
                BaseCommit: 03cd95bb0a3e7eab1823e35febdc88bae8d7b51dd171eb61a021f2557cf3df57
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: libvirt moby-engine qemu-kvm zsh
             LocalPackages: google-chrome-stable-76.0.3809.132-1.x86_64

  ostree://fedora:fedora/30/x86_64/silverblue
                   Version: 30.20190908.0 (2019-09-08T02:33:57Z)
                BaseCommit: 13e2ec82239a3d864ad400f3f375dce06c73dd2bbfc2124ae0279ab5f6c849af
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: libvirt moby-engine qemu-kvm zsh
             LocalPackages: google-chrome-stable-76.0.3809.132-1.x86_64
  • I cleaned up things: sudo rm -rf .local/share/containers/.
  • I created a fresh toolbox:
[yann@work-laptop ~]$ toolbox enter 
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/f30/fedora-toolbox:30 (500MB)? [y/N]: y
  • I tried to install jq from inside the toolbox:
⬢[yann@toolbox yann]$ sudo dnf install jq

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Fedora Modular 30 - x86_64                      2.1 MB/s | 2.7 MB     00:01    
Fedora Modular 30 - x86_64 - Updates            3.5 MB/s | 3.3 MB     00:00    
Fedora 30 - x86_64 - Updates             6.2 MB/s |  20 MB     00:03    :00 ETA
Fedora 30 - x86_64                             8.9 MB/s |  70 MB     00:07    
Dependencies resolved.
===============================================================================
 Package           Architecture   Version                Repository       Size
===============================================================================
Installing:
 jq                x86_64         1.6-2.fc30             fedora          168 k
Installing dependencies:
 oniguruma         x86_64         6.9.2-2.fc30           updates         193 k

Transaction Summary
===============================================================================
Install  2 Packages

Total download size: 361 k
Installed size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.9.2-2.fc30.x86_64.rpm       1.7 MB/s | 193 kB     00:00    
(2/2): jq-1.6-2.fc30.x86_64.rpm                1.3 MB/s | 168 kB     00:00    
-------------------------------------------------------------------------------
Total                                          216 kB/s | 361 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                       1/1 
  Installing       : oniguruma-6.9.2-2.fc30.x86_64                         1/2 
Error unpacking rpm package oniguruma-6.9.2-2.fc30.x86_64
  Installing       : jq-1.6-2.fc30.x86_64                                  2/2 
error: unpacking of archive failed on file /usr/lib/.build-id/1c/7588d6da78dd5888d79f988ab594f6b5abeeb5;5d74db39: cpio: utime
error: oniguruma-6.9.2-2.fc30.x86_64: install failed

Error unpacking rpm package jq-1.6-2.fc30.x86_64
  Verifying        : oniguruma-6.9.2-2.fc30.x86_64                         1/2 
  Verifying        : jq-1.6-2.fc30.x86_64                                  2/2 

Failed:
  oniguruma-6.9.2-2.fc30.x86_64              jq-1.6-2.fc30.x86_64             

Error: Transaction failed

From the /var/log/audit/audit.log file, I get:

type=AVC msg=audit(1567939700.806:344): avc:  denied  { setattr } for  pid=11197 comm="fuse-overlayfs" name="273" dev="proc" ino=141291 scontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=lnk_file permissive=0
type=AVC msg=audit(1567939700.809:345): avc:  denied  { setattr } for  pid=11197 comm="fuse-overlayfs" name="273" dev="proc" ino=141291 scontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=lnk_file permissive=0

Did I miss something or is this bug still present?

@rhatdan
Copy link
Member

rhatdan commented Sep 8, 2019

The AVC's indicate you need an updated version of container-selinux.

$ cat > /tmp/t


type=AVC msg=audit(1567939700.806:344): avc:  denied  { setattr } for  pid=11197 comm="fuse-overlayfs" name="273" dev="proc" ino=141291 scontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=lnk_file permissive=0
type=AVC msg=audit(1567939700.809:345): avc:  denied  { setattr } for  pid=11197 comm="fuse-overlayfs" name="273" dev="proc" ino=141291 scontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 tclass=lnk_file permissive=0
$ audit2allow  -i /tmp/t


#============= container_runtime_t ==============

#!!!! This avc is allowed in the current policy
allow container_runtime_t self:lnk_file setattr;

$ rpm -q container-selinux
container-selinux-2.114.0-1.git028ab00.fc30.noarch

@yann-soubeyrand
Copy link
Author

From the informations you gave above, I thought I had a fixed version:

[yann@work-laptop ~]$ rpm-ostree db list fedora:fedora/30/x86_64/testing/silverblue | grep container-selinux
 container-selinux-2:2.116.0-1.gitc5ef5ac.fc30.noarch

Which version does contain the fix? And where can I install it from?

@rhatdan
Copy link
Member

rhatdan commented Sep 8, 2019

It should be in that one, but I am wondering if it the modules is installed properly.

# rpm -qf /usr/share/selinux/packages/container.pp.bz2
container-selinux-2.114.0-1.git028ab00.fc30.noarch

# semodule -i /usr/share/selinux/packages/container.pp.bz2

@yann-soubeyrand
Copy link
Author

OK, sudo semodule -i /usr/share/selinux/packages/container.pp.bz2 solves the problem, thanks.

Can you keep us updated in this issue when the fix hits Silverblue 30 testing then stable?

@cgwalters
Copy link

Hm, you may have hit ostreedev/ostree#1026

@yann-soubeyrand
Copy link
Author

@cgwalters OK, if I understand correctly, this bug is not going to be fixed soon as it's not trivial, right? Is there a manual way to fix it (other than manually loading the policy after each reboot)?

@yann-soubeyrand
Copy link
Author

Answering myself: semodule command solves the problem permanently, thanks a lot @rhatdan ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants