No way to increase: Max open files

[Andrei-Stepanov]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I cannot build an app. Because of:

➜ podman run --rm -it --entrypoint /bin/bash centos/nodejs-10-centos7:latest
bash-4.2$ cat /proc/self/limits  | grep files
Max open files            1024                 1024                 files     

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.6.4

Output of podman info --debug:

  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 7a4f0dd7b20a3d4bf9ef3e5cbfac05606b08eac0'
  Distribution:
    distribution: '"centos"'
    version: "8"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 10022887424
  MemTotal: 16307253248
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 17179865088
  SwapTotal: 17179865088
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.2-2.git21fdece.module_el8.1.0+272+3e64ee36.x86_64
    Version: |-
      slirp4netns version 0.4.2+dev
      commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4
  uptime: 38m 52.7s
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /home/andrei/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-1.module_el8.1.0+272+3e64ee36.x86_64
      Version: |-
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  GraphRoot: /home/andrei/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /home/andrei/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.4-2.module_el8.1.0+272+3e64ee36.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

[root@localhost]~# cat /etc/security/limits.conf| grep nofile
#        - nofile - max number of open file descriptors
*               soft    nofile           1048576
*               hard    nofile           1048576

[mheon]

Are you running rootless, or as root?

[Andrei-Stepanov]

@mheon rootless.
The command is : npm install
To install javascript dependencies to build project.
No need root for this.

[Andrei-Stepanov]

Could be related to containers/buildah#2230

[mheon]

Can you manually adjust upwards with --ulimit nofile=1048576:1048576 and see if that works? I'm wondering if we can't adjust limits at all, or if we can adjust but rootless is not attempting to do so (I know that root Podman will automatically adjust to the higher possible).

[Andrei-Stepanov]

✗ podman run --ulimit nofile=1048576:1048576 --rm -it --entrypoint /bin/bash centos/nodejs-10-centos7:latest 
Error: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: operation not permitted\\\"\"": OCI runtime permission denied error

[Andrei-Stepanov]

➜ cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos2)/boot/vmlinuz-4.18.0-147.5.1.el8_1.x86_64 root=UUID=3d9a0fd2-0f01-4083-a3be-dead10385a24 ro crashkernel=auto resume=UUID=18fc7b98-4191-4623-b33c-795574c9fa2d rhgb quiet namespace.unpriv_enable=1 user_namespace.enable=1
➜  cat /etc/subgid
andrei:100000:65536
➜   cat /etc/subuid
andrei:100000:65536

[mheon]

And just to verify - can you do a cat /proc/self/limits as your user, but not in the container, and see that they're higher than the 1024:1024 that rootless Podman is giving you?

[Andrei-Stepanov]

In Centos 8 there are two different ways:
Gnome

➜  cat /proc/self/limits | grep open
Max open files            1024                 4096                 files  

Console: Ctrl-Alt-F3:

BUT! In both cases INSIDE container is 1024.

[Andrei-Stepanov]

Okay, seems found solution! @mheon the issue can be closed.
Now I need to find how to increase limits for Gnome session.
Thank you for your help!
@mheon ++

[mheon]

Glad to help!

[sefroberg]

Okay, seems found solution! @mheon the issue can be closed.
Now I need to find how to increase limits for Gnome session.
Thank you for your help!
@mheon ++

@Andrei-Stepanov
What was the fix for you?

[Andrei-Stepanov]

Hi, I do not remember exactly.
The link I saved for solution was: https://success.docker.com/article/user-namespace-runtime-error
But they removed that page.
I will put related links that maybe would help:

I have a note about 2 packages:

yum install https://cbs.centos.org/kojifiles/packages/libvarlink/12/1.el7/x86_64/libvarlink-util-12-1.el7.x86_64.rpm
shadow-utils-4.6-5.el7.x86_64 (I believe this has support for /etc/subuid  and  /etc/subgid)

Google for: "unpriv_enable" docker centos
I know this is about podman, the root cause is the same

More links
https://medium.com/@Mark.io/simple-rootless-containers-with-runc-on-centos-redhat-f9230f74b88b
https://docs.docker.com/engine/security/userns-remap/#about-remapping-and-subordinate-user-and-group-ids
https://gist.github.com/mjuric/c519d470eac60b08de5ed735ff5a2ef9

[bmaupin]

As best as I can tell, the values of podman --ulimit can't be set any higher than the values of the user running podman.

Here's what I did to fix this (this was on a RHEL 7 machine):

1. On the host machine, check the limits of the current user

   $ cat /proc/self/limits | grep open
   Max open files            1024                 4096                 files
   

2. (Optional) View the limits in a container

   $ podman run --rm -it registry.access.redhat.com/ubi8/ubi cat /proc/self/limits | grep open
   Max open files            1024                 1024                 files
   

3. Add a new limits.conf file to update the limits

   sudo sh -c "echo '* soft     nofile         65535
   * hard     nofile         65535' > /etc/security/limits.d/30-nofile.conf"
   

4. Log out and log back in

5. Check the user limits again

   $ cat /proc/self/limits | grep open
   Max open files            65535                65535                files
   

6. Now you should be able to run podman with --ulimit, e.g.

   $ podman run --ulimit nofile=65535:65535 --rm -it registry.access.redhat.com/ubi8/ubi cat /proc/self/limits | grep open
   Max open files            65535                65535                files
   

   ⚠ Make sure whatever values you use for ulimit are no higher than the values you set for the user (e.g. in this example, max open files for the user was set to 65535 so that's what was used for podman --ulimit)