Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Podman update doesn't have effect on memory limit #18621

Closed
rdbisme opened this issue May 18, 2023 · 2 comments · Fixed by #18630
Closed

Podman update doesn't have effect on memory limit #18621

rdbisme opened this issue May 18, 2023 · 2 comments · Fixed by #18630
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@rdbisme
Copy link

rdbisme commented May 18, 2023 •
edited
Loading

Issue Description

`podman inspect minecraft` 
[
  {
    "Id": "39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793",
    "Created": "2023-05-07T21:09:35.001003491+02:00",
    "Path": "/usr/local/bin/entrypoint-demoter",
    "Args": [
      "--match",
      "/data",
      "--debug",
      "--stdin-on-term",
      "stop",
      "/opt/bedrock-entry.sh"
    ],
    "State": {
      "OciVersion": "1.1.0-rc.1",
      "Status": "running",
      "Running": true,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 1505982,
      "ConmonPid": 1505980,
      "ExitCode": 0,
      "Error": "container is stopped",
      "StartedAt": "2023-05-18T18:37:27.01371332+02:00",
      "FinishedAt": "2023-05-18T18:37:26.528386074+02:00",
      "Health": {
        "Status": "healthy",
        "FailingStreak": 0,
        "Log": [
          {
            "Start": "2023-05-18T18:44:10.126396963+02:00",
            "End": "2023-05-18T18:44:10.288643512+02:00",
            "ExitCode": 0,
            "Output": ""
          },
          {
            "Start": "2023-05-18T18:44:41.136608901+02:00",
            "End": "2023-05-18T18:44:41.297920664+02:00",
            "ExitCode": 0,
            "Output": ""
          },
          {
            "Start": "2023-05-18T18:45:12.132404987+02:00",
            "End": "2023-05-18T18:45:12.373945626+02:00",
            "ExitCode": 0,
            "Output": ""
          },
          {
            "Start": "2023-05-18T18:45:43.135531846+02:00",
            "End": "2023-05-18T18:45:43.291669896+02:00",
            "ExitCode": 0,
            "Output": ""
          },
          {
            "Start": "2023-05-18T18:46:14.134814568+02:00",
            "End": "2023-05-18T18:46:14.309708567+02:00",
            "ExitCode": 0,
            "Output": ""
          }
        ]
      },
      "CgroupPath": "/user.slice/user-1006.slice/[email protected]/user.slice/libpod-39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793.scope",
      "CheckpointedAt": "0001-01-01T00:00:00Z",
      "RestoredAt": "0001-01-01T00:00:00Z"
    },
    "Image": "ab6ac061a5bb8b7c721b6ee2768b8500e8dc6ac2147686749a22c02cfcf3b0ba",
    "ImageDigest": "sha256:74df3f9fd03c582822a8a45f9d3a83d1eaebe440950aa377102926a7133db3a0",
    "ImageName": "docker.io/itzg/minecraft-bedrock-server:latest",
    "Rootfs": "",
    "Pod": "",
    "ResolvConfPath": "/tmp/containers-user-1006/containers/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/resolv.conf",
    "HostnamePath": "/tmp/containers-user-1006/containers/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/hostname",
    "HostsPath": "/tmp/containers-user-1006/containers/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/hosts",
    "StaticDir": "/home/minecraft/.local/share/containers/storage/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata",
    "OCIConfigPath": "/home/minecraft/.local/share/containers/storage/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/config.json",
    "OCIRuntime": "crun",
    "ConmonPidFile": "/tmp/containers-user-1006/containers/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/conmon.pid",
    "PidFile": "/tmp/containers-user-1006/containers/overlay-containers/39c2b66ed391372921dc55a467f6f0791e0997bcec3e2e97ed5e688b49087793/userdata/pidfile",
    "Name": "minecraft",
    "RestartCount": 0,
    "Driver": "overlay",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "EffectiveCaps": [
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_NET_BIND_SERVICE",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "BoundingCaps": [
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FOWNER",
      "CAP_FSETID",
      "CAP_KILL",
      "CAP_NET_BIND_SERVICE",
      "CAP_SETFCAP",
      "CAP_SETGID",
      "CAP_SETPCAP",
      "CAP_SETUID",
      "CAP_SYS_CHROOT"
    ],
    "ExecIDs": [],
    "GraphDriver": {
      "Name": "overlay",
      "Data": {
        "LowerDir": "/home/minecraft/.local/share/containers/storage/overlay/bee7d59ab1ee7c38f0d31ae823794e81621a838c263762a31a6c75c61931ea2e/diff:/home/minecraft/.local/share/containers/storage/overlay/bcbc9a367cc4957f6b2159425cda6384a0e55bab715b2fa84eb208df3428144a/diff:/home/minecraft/.local/share/containers/storage/overlay/9e4cedd1cb867db9cd8aa220adb577c181d7b0f9be24f799fe5ef20606cb43ea/diff:/home/minecraft/.local/share/containers/storage/overlay/b199a94d0289655bbc1431e4493104028e71dbc2ca5d2690062771b038145abf/diff:/home/minecraft/.local/share/containers/storage/overlay/bfc3e8b61003b6dfed4a446b37d28ec457dbb155ac18755b7a415ba3c1aa553f/diff:/home/minecraft/.local/share/containers/storage/overlay/e50926437b5b300185f8c6a4e79008eaaa908ca95ba7f85f5db68db938fbe939/diff:/home/minecraft/.local/share/containers/storage/overlay/0e86d5530a65594e38940d00b05e787fdc24308860dc9f6c21af6f88f614f76c/diff:/home/minecraft/.local/share/containers/storage/overlay/0bac64ab35d0f61b075ff1d0116841fb070e9ba9e08a7e0b1dcaab5275afa129/diff:/home/minecraft/.local/share/containers/storage/overlay/c55bf16b46c5a897bb324b84995517c9a8d7e99b28014b2fe4274b319f1d6259/diff:/home/minecraft/.local/share/containers/storage/overlay/f348e8aef39c0d1003b62310664f504ab2eb37fec581ae170939d86cd38644cc/diff:/home/minecraft/.local/share/containers/storage/overlay/68254d5c1ea53012f81cdd37d95503fdab0a1ca99fcc39ef15aae83be16843e2/diff:/home/minecraft/.local/share/containers/storage/overlay/82f11cfd33390b488677f830e52d241dad2642c1690ff11f81e0dcfdd31e434e/diff:/home/minecraft/.local/share/containers/storage/overlay/ec09eb83ea031896df916feb3a61cefba9facf449c8a55d88667927538dca2b4/diff",
        "MergedDir": "/home/minecraft/.local/share/containers/storage/overlay/64bf4bb33c5c55ccebd9c7af32121966c4a2370a9edb3b803c7c40892fb5da6f/merged",
        "UpperDir": "/home/minecraft/.local/share/containers/storage/overlay/64bf4bb33c5c55ccebd9c7af32121966c4a2370a9edb3b803c7c40892fb5da6f/diff",
        "WorkDir": "/home/minecraft/.local/share/containers/storage/overlay/64bf4bb33c5c55ccebd9c7af32121966c4a2370a9edb3b803c7c40892fb5da6f/work"
      }
    },
    "Mounts": [
      {
        "Type": "bind",
        "Source": "/home/minecraft/.local/share/containers/volumes/minecraft/data",
        "Destination": "/data",
        "Driver": "",
        "Mode": "",
        "Options": [
          "rbind"
        ],
        "RW": true,
        "Propagation": "rprivate"
      }
    ],
    "Dependencies": [],
    "NetworkSettings": {
      "EndpointID": "",
      "Gateway": "",
      "IPAddress": "",
      "IPPrefixLen": 0,
      "IPv6Gateway": "",
      "GlobalIPv6Address": "",
      "GlobalIPv6PrefixLen": 0,
      "MacAddress": "",
      "Bridge": "",
      "SandboxID": "",
      "HairpinMode": false,
      "LinkLocalIPv6Address": "",
      "LinkLocalIPv6PrefixLen": 0,
      "Ports": {
        "19132/udp": [
          {
            "HostIp": "",
            "HostPort": "19132"
          }
        ]
      },
      "SandboxKey": "/run/user/1006/netns/netns-19a274ae-8895-0e0c-61bc-0ee45c50800d"
    },
    "Namespace": "",
    "IsInfra": false,
    "IsService": false,
    "Config": {
      "Hostname": "39c2b66ed391",
      "Domainname": "",
      "User": "",
      "AttachStdin": false,
      "AttachStdout": false,
      "AttachStderr": false,
      "Tty": false,
      "OpenStdin": false,
      "StdinOnce": false,
      "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "TERM=xterm",
        "container=podman",
        "VERSION=LATEST",
        "SERVER_PORT=19132",
        "EULA=TRUE",
        "ALLOW_LIST_USERS=Tidusuper91,Petitepomme77",
        "HOME=/root",
        "HOSTNAME=39c2b66ed391"
      ],
      "Cmd": null,
      "Image": "docker.io/itzg/minecraft-bedrock-server:latest",
      "Volumes": null,
      "WorkingDir": "/data",
      "Entrypoint": "/usr/local/bin/entrypoint-demoter --match /data --debug --stdin-on-term stop /opt/bedrock-entry.sh",
      "OnBuild": null,
      "Labels": {
        "manymine.enable": "true",
        "org.opencontainers.image.created": "2023-04-10T18:31:25.730Z",
        "org.opencontainers.image.description": "Containerized Minecraft Bedrock Dedicated Server with selectable version",
        "org.opencontainers.image.licenses": "MIT",
        "org.opencontainers.image.revision": "64f9c3a250fd529dd6f5678d386c524f61672f11",
        "org.opencontainers.image.source": "https://github.com/itzg/docker-minecraft-bedrock-server",
        "org.opencontainers.image.title": "docker-minecraft-bedrock-server",
        "org.opencontainers.image.url": "https://github.com/itzg/docker-minecraft-bedrock-server",
        "org.opencontainers.image.version": "master"
      },
      "Annotations": {
        "io.container.manager": "libpod",
        "org.opencontainers.image.stopSignal": "15"
      },
      "StopSignal": 15,
      "Healthcheck": {
        "Test": [
          "CMD-SHELL",
          "/usr/local/bin/mc-monitor status-bedrock --host 127.0.0.1 --port $SERVER_PORT"
        ],
        "StartPeriod": 60000000000,
        "Interval": 30000000000,
        "Timeout": 30000000000
      },
      "HealthcheckOnFailureAction": "none",
      "CreateCommand": [
        "podman",
        "container",
        "run",
        "--name",
        "minecraft",
        "--cpus",
        "1",
        "--memory",
        "768M",
        "--env",
        "EULA=TRUE",
        "--env",
        "ALLOW_LIST_USERS=Tidusuper91,Petitepomme77",
        "--publish",
        "19132:19132/udp",
        "--volume",
        "/home/minecraft/.local/share/containers/volumes/minecraft/data:/data:z",
        "--detach=True",
        "docker.io/itzg/minecraft-bedrock-server"
      ],
      "Umask": "0022",
      "Timeout": 0,
      "StopTimeout": 10,
      "Passwd": true,
      "sdNotifyMode": "container"
    },
    "HostConfig": {
      "Binds": [
        "/home/minecraft/.local/share/containers/volumes/minecraft/data:/data:rw,rprivate,rbind"
      ],
      "CgroupManager": "systemd",
      "CgroupMode": "private",
      "ContainerIDFile": "",
      "LogConfig": {
        "Type": "journald",
        "Config": null,
        "Path": "",
        "Tag": "",
        "Size": "0B"
      },
      "NetworkMode": "slirp4netns",
      "PortBindings": {
        "19132/udp": [
          {
            "HostIp": "",
            "HostPort": "19132"
          }
        ]
      },
      "RestartPolicy": {
        "Name": "",
        "MaximumRetryCount": 0
      },
      "AutoRemove": false,
      "VolumeDriver": "",
      "VolumesFrom": null,
      "CapAdd": [],
      "CapDrop": [],
      "Dns": [],
      "DnsOptions": [],
      "DnsSearch": [],
      "ExtraHosts": [],
      "GroupAdd": [],
      "IpcMode": "shareable",
      "Cgroup": "",
      "Cgroups": "default",
      "Links": null,
      "OomScoreAdj": 0,
      "PidMode": "private",
      "Privileged": false,
      "PublishAllPorts": false,
      "ReadonlyRootfs": false,
      "SecurityOpt": [],
      "Tmpfs": {},
      "UTSMode": "private",
      "UsernsMode": "",
      "ShmSize": 65536000,
      "Runtime": "oci",
      "ConsoleSize": [
        0,
        0
      ],
      "Isolation": "",
      "CpuShares": 0,
      "Memory": 805306368,
      "NanoCpus": 1000000000,
      "CgroupParent": "user.slice",
      "BlkioWeight": 0,
      "BlkioWeightDevice": null,
      "BlkioDeviceReadBps": null,
      "BlkioDeviceWriteBps": null,
      "BlkioDeviceReadIOps": null,
      "BlkioDeviceWriteIOps": null,
      "CpuPeriod": 100000,
      "CpuQuota": 100000,
      "CpuRealtimePeriod": 0,
      "CpuRealtimeRuntime": 0,
      "CpusetCpus": "",
      "CpusetMems": "",
      "Devices": [],
      "DiskQuota": 0,
      "KernelMemory": 0,
      "MemoryReservation": 0,
      "MemorySwap": 1610612736,
      "MemorySwappiness": -1,
      "OomKillDisable": false,
      "PidsLimit": 2048,
      "Ulimits": [
        {
          "Name": "RLIMIT_NOFILE",
          "Soft": 524288,
          "Hard": 524288
        },
        {
          "Name": "RLIMIT_NPROC",
          "Soft": 127896,
          "Hard": 127896
        }
      ],
      "CpuCount": 0,
      "CpuPercent": 0,
      "IOMaximumIOps": 0,
      "IOMaximumBandwidth": 0,
      "CgroupConf": null
    }
  }
]
ID            NAME        CPU %       MEM USAGE / LIMIT  MEM %       NET IO             BLOCK IO    PIDS        CPU TIME    AVG CPU %
39c2b66ed391  minecraft   1.35%       143.9MB / 805.3MB  17.86%      88.36kB / 3.408kB  0B / 0B     29          10.265778s  1.52%

Then I want to update the memory limit:

podman update --memory="2g" minecraft.

Still podman stats shows:

ID            NAME        CPU %       MEM USAGE / LIMIT  MEM %       NET IO             BLOCK IO    PIDS        CPU TIME    AVG CPU %
39c2b66ed391  minecraft   1.35%       143.9MB / 805.3MB  17.86%      88.36kB / 3.408kB  0B / 0B     29          11.194684s  1.51%

Steps to reproduce the issue

Steps to reproduce the issue

  1. podman update --memory="2g" <container>
  2. podman stats

Describe the results you received

The updated memory limit doesn't seem to have any effect.

Describe the results you expected

The memory limit would change and podman stats would show that.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.7-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 99.31
    systemPercent: 0.28
    userPercent: 0.41
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: gagazet
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 6.1.28-1-lts
  linkmode: dynamic
  logDriver: journald
  memFree: 10015350784
  memTotal: 33555595264
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8.3-2
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1006/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1006/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 139h 13m 37.00s (Approximately 5.79 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/minecraft/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/minecraft/.local/share/containers/storage
  graphRootAllocated: 2151454212096
  graphRootUsed: 8888123392
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1006/containers
  transientStore: false
  volumePath: /home/minecraft/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1684095102
  BuiltTime: Sun May 14 22:11:42 2023
  GitCommit: 75e3c12579d391b81d871fd1cded6cf0d043550a-dirty
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
@rdbisme rdbisme added the kind/bug Categorizes issue or PR as related to a bug. label May 18, 2023
@giuseppe
Copy link
Member

I think we are using the limit from the container configuration instead of reading it from the cgroup.

The container update operation is a transient operation that only affects the running container and it won't affect the container configuration. If you restart your container, the setting will be lost.

More details here: #15067

@giuseppe giuseppe mentioned this issue May 19, 2023
Merged
@giuseppe
Copy link
Member

I've opened a PR to fix it: #18630

It also needs containers/crun#1217 in crun

giuseppe added a commit to giuseppe/libpod that referenced this issue May 19, 2023
@giuseppe
stats: get mem limit from the cgroup
7c53a46 
b25b330 introduced this behaviour.

It was fine at the time because we didn't support "container update",
so the limit could not be changed at runtime.  Since it is not
possible to change the memory limit at runtime, read the limit as
reported from the cgroup.

containers/crun#1217 is required for crun.

Closes: containers#18621

Signed-off-by: Giuseppe Scrivano <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

stats: get mem limit from the cgroup giuseppe/libpod
2 participants
@giuseppe @rdbisme