Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Process oom_score_adj is set to -1000 after termination by oom-killer #20765

Closed
hanneshauer opened this issue Nov 23, 2023 · 6 comments · Fixed by containers/conmon#464
Closed

Process oom_score_adj is set to -1000 after termination by oom-killer #20765

hanneshauer opened this issue Nov 23, 2023 · 6 comments · Fixed by containers/conmon#464
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@hanneshauer
Copy link

hanneshauer commented Nov 23, 2023 •
edited
Loading

Issue description

When a container is restarted after its process is terminated by oom-killer due to exceeding memory limits, the process in the restarted is assigned an oom_score_adj value of -1000.
The process is therefore unkillable by oom-killer after the first restart, which keeps the container running even when exceeding memory limits agian, and triggers excessive logging.

Steps to reproduce the issue

Container and systemd config using podman kube play:

  1. Create /etc/containers/systemd/stress.kube:
[Kube]
Yaml=/tmp/stress.kube.yaml

[Unit]
Description=Stresstest

[Service]
OOMScoreAdjust=666 # Optional, added for debugging purposes
  1. Create Pod configuration in /tmp/stress.kube.yaml:
---
apiVersion: apps/v1
kind: Pod
metadata:
  name: stresstest
spec:
  containers:
  - name: stresstest-1
    image: docker.io/progrium/stress
    args: ["--vm", "8", "--vm-bytes", "5120M"]
    resources:
      limits:
        memory: "1000M"
  1. Reload systemd daemon and start service: systemd daemon-reload && system start stress
  2. After the container exceeds its memory limit and is restarted, check the oom_score_adj value for the new process: cat /proc/<pid>/oom_score_adj

Describe the results you received

A container that was killed due to OOM cannot be killed automatically because its process is assigned an oom_score_adj of -1000. The container can manually be stopped or restarted however. The syslog indicates that the kernels oom-killer cannot find a process to kill to free more memory ("Out of memory and no killable processes...")

/var/log/messages

The syslog shows that the containers process has been assigned the oom_score_adj as specified in the systemd unit when first started. oom-killer is invoked and terminates the process successfully. Podman registers that the container died and restarts it. Upon again reaching the memory limit, oom-killer only finds processes with a oom_score_adj of -1000 and is therefore unable to kill any process.

Nov 23 15:36:19 localhost kernel: stress invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=666
Nov 23 15:36:19 localhost kernel: CPU: 1 PID: 5074 Comm: stress Kdump: loaded Tainted: G           OE    --------  ---  5.14.0-285.el9.x86_64 containers/crun#1
Nov 23 15:36:19 localhost kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Nov 23 15:36:19 localhost kernel: Call Trace:
Nov 23 15:36:19 localhost kernel: <TASK>
Nov 23 15:36:19 localhost kernel: dump_stack_lvl+0x34/0x48
Nov 23 15:36:19 localhost kernel: dump_header+0x4a/0x201
Nov 23 15:36:19 localhost kernel: oom_kill_process.cold+0xb/0x10
Nov 23 15:36:19 localhost kernel: out_of_memory+0xed/0x2e0
Nov 23 15:36:19 localhost kernel: mem_cgroup_out_of_memory+0x13a/0x150
Nov 23 15:36:19 localhost kernel: try_charge_memcg+0x6df/0x7a0
Nov 23 15:36:19 localhost kernel: ? __alloc_pages+0xe6/0x230
Nov 23 15:36:19 localhost kernel: charge_memcg+0x9f/0x130
Nov 23 15:36:19 localhost kernel: __mem_cgroup_charge+0x29/0x80
Nov 23 15:36:19 localhost kernel: do_anonymous_page+0x100/0x4f0
Nov 23 15:36:19 localhost kernel: __handle_mm_fault+0x401/0x730
Nov 23 15:36:19 localhost kernel: handle_mm_fault+0xc5/0x2a0
Nov 23 15:36:19 localhost kernel: do_user_addr_fault+0x1bb/0x690
Nov 23 15:36:19 localhost kernel: exc_page_fault+0x62/0x150
Nov 23 15:36:19 localhost kernel: asm_exc_page_fault+0x22/0x30
Nov 23 15:36:19 localhost kernel: RIP: 0033:0x402ae0
Nov 23 15:36:19 localhost kernel: Code: 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f 1e 02 00 00 48 85 ed 48 89 d8 7e 1b 66 2e 0f 1f 84 00 00 00 00 00 <c6> 00 5a 4c 01 f8 48 89 c2 48 29 da 48 39 d5 7f ef 49 83 fc 00 0f
Nov 23 15:36:19 localhost kernel: RSP: 002b:00007ffcedf1bcd0 EFLAGS: 00010206
Nov 23 15:36:19 localhost kernel: RAX: 00007ff55ebc1010 RBX: 00007ff54a475010 RCX: 0000000000000021
Nov 23 15:36:19 localhost kernel: RDX: 000000001474c000 RSI: 000000000000002c RDI: 00007ff68ad610de
Nov 23 15:36:19 localhost kernel: RBP: 0000000140000000 R08: 00007ff68a5edc40 R09: 0000000000000000
Nov 23 15:36:19 localhost kernel: R10: 00007ff68a8346a0 R11: 0000000000000000 R12: ffffffffffffffff
Nov 23 15:36:19 localhost kernel: R13: 0000000000000003 R14: fffffffffffff000 R15: 0000000000001000
Nov 23 15:36:19 localhost kernel: </TASK>
Nov 23 15:36:19 localhost kernel: memory: usage 976560kB, limit 976560kB, failcnt 9918
Nov 23 15:36:19 localhost kernel: swap: usage 2148348kB, limit 9007199254740988kB, failcnt 0
Nov 23 15:36:19 localhost kernel: Memory cgroup stats for /machine.slice/machine-libpod_pod_91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2.slice/libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope:
Nov 23 15:36:19 localhost kernel: anon 985669632#012file 0#012kernel 7258112#012kernel_stack 131072#012pagetables 6647808#012percpu 7024#012sock 0#012vmalloc 12288#012shmem 0#012zswap 0#012zswapped 0#012file_mapped 0#012file_dirty 0#012file_writeback 20480#012swapcached 7024640#012anon_thp 0#012file_thp 0#012shmem_thp 0#012inactive_anon 991932416#012active_anon 253952#012inactive_file 0#012active_file 0#012unevictable 0#012slab_reclaimable 82456#012slab_unreclaimable 222848#012slab 305304#012workingset_refault_anon 0#012workingset_refault_file 0#012workingset_activate_anon 0#012workingset_activate_file 0#012workingset_restore_anon 0#012workingset_restore_file 0#012workingset_nodereclaim 0#012pgscan 1751966#012pgsteal 535371#012pgscan_kswapd 0#012pgscan_direct 1751966#012pgsteal_kswapd 0#012pgsteal_direct 535371#012pgfault 538834#012pgmajfault 0#012pgrefill 0#012pgactivate 61#012pgdeactivate 0#012pglazyfree 0#012pglazyfreed 0#012zswpin 0#012zswpout 0#012thp_fault_alloc 468#012thp_collapse_alloc 0
Nov 23 15:36:19 localhost kernel: Tasks state (memory values in pages):
Nov 23 15:36:19 localhost kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 23 15:36:19 localhost kernel: [   5072]     0  5072     1830      205    49152       15           666 stress
Nov 23 15:36:19 localhost kernel: [   5074]     0  5074  1312551    30540   724992    53488           666 stress
Nov 23 15:36:19 localhost kernel: [   5075]     0  5075  1312551    31341   749568    55476           666 stress
Nov 23 15:36:19 localhost kernel: [   5076]     0  5076  1312551    25848   778240    64761           666 stress
Nov 23 15:36:19 localhost kernel: [   5077]     0  5077  1312551    32952   745472    53173           666 stress
Nov 23 15:36:19 localhost kernel: [   5078]     0  5078  1312551    28655   765952    60517           666 stress
Nov 23 15:36:19 localhost kernel: [   5079]     0  5079  1312551    31855   839680    66144           666 stress
Nov 23 15:36:19 localhost kernel: [   5080]     0  5080  1312551    31170  1220608   114403           666 stress
Nov 23 15:36:19 localhost kernel: [   5081]     0  5081  1312551    30032   847872    69147           666 stress
Nov 23 15:36:19 localhost kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=container,mems_allowed=0,oom_memcg=/machine.slice/machine-libpod_pod_91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2.slice/libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope,task_memcg=/machine.slice/machine-libpod_pod_91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2.slice/libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope/container,task=stress,pid=5080,uid=0
Nov 23 15:36:19 localhost kernel: Memory cgroup out of memory: Killed process 5080 (stress) total-vm:5250204kB, anon-rss:123668kB, file-rss:1012kB, shmem-rss:0kB, UID:0 pgtables:1192kB oom_score_adj:666
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: A process of this unit has been killed by the OOM killer.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5072 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5074 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5075 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5076 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5077 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5078 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5079 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5080 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Killing process 5081 (stress) with signal SIGKILL.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Failed with result 'oom-kill'.
Nov 23 15:36:19 localhost systemd[1]: libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope: Consumed 3.359s CPU time.
Nov 23 15:36:19 localhost podman[5101]: 2023-11-23 15:36:19.57911526 +0000 UTC m=+0.012576171 container died ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351 (image=docker.io/progrium/stress:latest, name=stresstest-stresstest-1, PODMAN_SYSTEMD_UNIT=stress.service)
Nov 23 15:36:19 localhost podman[5101]: 2023-11-23 15:36:19.581995874 +0000 UTC m=+0.015456785 container restart ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351 (image=docker.io/progrium/stress:latest, name=stresstest-stresstest-1, pod_id=91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2, PODMAN_SYSTEMD_UNIT=stress.service)
Nov 23 15:36:19 localhost systemd[1]: tmp-crun.HJ1p1x.mount: Deactivated successfully.
Nov 23 15:36:19 localhost systemd[1]: Started libpod-conmon-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope.
Nov 23 15:36:19 localhost systemd[1]: tmp-crun.y8BBWW.mount: Deactivated successfully.
Nov 23 15:36:19 localhost systemd[1]: Started libcrun container.
Nov 23 15:36:19 localhost podman[5101]: 2023-11-23 15:36:19.612782766 +0000 UTC m=+0.046243686 container init ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351 (image=docker.io/progrium/stress:latest, name=stresstest-stresstest-1, pod_id=91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2, PODMAN_SYSTEMD_UNIT=stress.service)
Nov 23 15:36:19 localhost podman[5101]: 2023-11-23 15:36:19.614727431 +0000 UTC m=+0.048188337 container start ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351 (image=docker.io/progrium/stress:latest, name=stresstest-stresstest-1, pod_id=91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2, PODMAN_SYSTEMD_UNIT=stress.service)
Nov 23 15:36:25 localhost kernel: stress invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-1000
Nov 23 15:36:25 localhost kernel: CPU: 0 PID: 5117 Comm: stress Kdump: loaded Tainted: G           OE    --------  ---  5.14.0-285.el9.x86_64 containers/crun#1
Nov 23 15:36:25 localhost kernel: Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Nov 23 15:36:25 localhost kernel: Call Trace:
Nov 23 15:36:25 localhost kernel: <TASK>
Nov 23 15:36:25 localhost kernel: dump_stack_lvl+0x34/0x48
Nov 23 15:36:25 localhost kernel: dump_header+0x4a/0x201
Nov 23 15:36:25 localhost kernel: out_of_memory.cold+0xa/0x7e
Nov 23 15:36:25 localhost kernel: mem_cgroup_out_of_memory+0x13a/0x150
Nov 23 15:36:25 localhost kernel: try_charge_memcg+0x6df/0x7a0
Nov 23 15:36:25 localhost kernel: ? __alloc_pages+0xe6/0x230
Nov 23 15:36:25 localhost kernel: charge_memcg+0x9f/0x130
Nov 23 15:36:25 localhost kernel: __mem_cgroup_charge+0x29/0x80
Nov 23 15:36:25 localhost kernel: do_anonymous_page+0x100/0x4f0
Nov 23 15:36:25 localhost kernel: __handle_mm_fault+0x401/0x730
Nov 23 15:36:25 localhost kernel: handle_mm_fault+0xc5/0x2a0
Nov 23 15:36:25 localhost kernel: do_user_addr_fault+0x1bb/0x690
Nov 23 15:36:25 localhost kernel: exc_page_fault+0x62/0x150
Nov 23 15:36:25 localhost kernel: asm_exc_page_fault+0x22/0x30
Nov 23 15:36:25 localhost kernel: RIP: 0033:0x402ae0
Nov 23 15:36:25 localhost kernel: Code: 8b 54 24 0c 31 c0 85 d2 0f 94 c0 89 04 24 41 83 fd 02 0f 8f 1e 02 00 00 48 85 ed 48 89 d8 7e 1b 66 2e 0f 1f 84 00 00 00 00 00 <c6> 00 5a 4c 01 f8 48 89 c2 48 29 da 48 39 d5 7f ef 49 83 fc 00 0f
Nov 23 15:36:25 localhost kernel: RSP: 002b:00007ffc53841310 EFLAGS: 00010206
Nov 23 15:36:25 localhost kernel: RAX: 00007faf50ae5010 RBX: 00007faf3c585010 RCX: 0000000000000021
Nov 23 15:36:25 localhost kernel: RDX: 0000000014560000 RSI: 000000000000002c RDI: 00007fb07ce711a0
Nov 23 15:36:25 localhost kernel: RBP: 0000000140000000 R08: 00007fb07c6fdc40 R09: 0000000000000000
Nov 23 15:36:25 localhost kernel: R10: 00007fb07c9446a0 R11: 0000000000000000 R12: ffffffffffffffff
Nov 23 15:36:25 localhost kernel: R13: 0000000000000003 R14: fffffffffffff000 R15: 0000000000001000
Nov 23 15:36:25 localhost kernel: </TASK>
Nov 23 15:36:25 localhost kernel: memory: usage 976560kB, limit 976560kB, failcnt 9910
Nov 23 15:36:25 localhost kernel: swap: usage 2148348kB, limit 9007199254740988kB, failcnt 0
Nov 23 15:36:25 localhost kernel: Memory cgroup stats for /machine.slice/machine-libpod_pod_91699c4e43f2b1b3691c25ce4920a8d9675e6e4d4d49855a2f9d07c9346066c2.slice/libpod-ac7c09aae3d09460c9d4b36bb2d1178db96d2a61d3f9de863711b2d478ad6351.scope:
Nov 23 15:36:25 localhost kernel: anon 984326144#012file 0#012kernel 7315456#012kernel_stack 131072#012pagetables 6729728#012percpu 7024#012sock 0#012vmalloc 12288#012shmem 0#012zswap 0#012zswapped 0#012file_mapped 0#012file_dirty 0#012file_writeback 24576#012swapcached 8261632#012anon_thp 0#012file_thp 0#012shmem_thp 0#012inactive_anon 991576064#012active_anon 278528#012inactive_file 0#012active_file 0#012unevictable 0#012slab_reclaimable 60984#012slab_unreclaimable 222848#012slab 283832#012workingset_refault_anon 12#012workingset_refault_file 0#012workingset_activate_anon 6#012workingset_activate_file 0#012workingset_restore_anon 0#012workingset_restore_file 0#012workingset_nodereclaim 0#012pgscan 1748214#012pgsteal 535059#012pgscan_kswapd 0#012pgscan_direct 1748214#012pgsteal_kswapd 0#012pgsteal_direct 535059#012pgfault 535965#012pgmajfault 2#012pgrefill 0#012pgactivate 61#012pgdeactivate 0#012pglazyfree 0#012pglazyfreed 0#012zswpin 0#012zswpout 0#012thp_fault_alloc 473#012thp_collapse_alloc 0
Nov 23 15:36:25 localhost kernel: Tasks state (memory values in pages):
Nov 23 15:36:25 localhost kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov 23 15:36:25 localhost kernel: [   5113]     0  5113     1830      416    53248       15         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5115]     0  5115  1312551    29474   724992    52945         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5116]     0  5116  1312551    30082   696320    48990         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5117]     0  5117  1312551    32987   733184    50575         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5118]     0  5118  1312551    31480   991232    84107         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5119]     0  5119  1312551    29928   618496    39203         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5120]     0  5120  1312551    31459   610304    36764         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5121]     0  5121  1312551    26584  1019904    93038         -1000 stress
Nov 23 15:36:25 localhost kernel: [   5122]     0  5122  1312551    30169  1355776   131487         -1000 stress
Nov 23 15:36:25 localhost kernel: Out of memory and no killable processes...
Nov 23 15:36:25 localhost kernel: Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
Nov 23 15:36:25 localhost kernel: stress invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-1000
Nov 23 15:36:25 localhost kernel: CPU: 0 PID: 5117 Comm: stress Kdump: loaded Tainted: G           OE    --------  ---  5.14.0-285.el9.x86_64 containers/crun#1

podman container inspect

After restart triggered by oom-killer ("HostConfig.OomScoreAdj": 0):

[
     {
          "Id": "fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02",
          "Created": "2023-11-23T16:47:58.771520818Z",
          "Path": "/usr/bin/stress",
          "Args": [
               "--verbose",
               "--vm",
               "8",
               "--vm-bytes",
               "5120M"
          ],
          "State": {
               "OciVersion": "1.1.0+dev",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 5096,
               "ConmonPid": 5094,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2023-11-23T16:47:59.038994811Z",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CgroupPath": "/machine.slice/machine-libpod_pod_881e17054a4d9459300178c0893d5ca44e2e5853efa04f021328f0a6832c8571.slice/libpod-fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "db646a8f40875981809f754e28a3834e856727b12e7662dad573b6b243e3fba4",
          "ImageDigest": "sha256:e34d56d60f5caae79333cee395aae93b74791d50e3841986420d23c2ee4697bf",
          "ImageName": "docker.io/progrium/stress:latest",
          "Rootfs": "",
          "Pod": "881e17054a4d9459300178c0893d5ca44e2e5853efa04f021328f0a6832c8571",
          "ResolvConfPath": "/run/containers/storage/overlay-containers/25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b/userdata/resolv.conf",
          "HostnamePath": "/run/containers/storage/overlay-containers/fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02/userdata/hostname",
          "HostsPath": "/run/containers/storage/overlay-containers/25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b/userdata/hosts",
          "StaticDir": "/var/lib/containers/storage/overlay-containers/fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02/userdata",
          "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/containers/storage/overlay-containers/fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02/userdata/conmon.pid",
          "PidFile": "/run/containers/storage/overlay-containers/fff80b29f0fbbfb7f09f160d206954aaaa6b459d31362fb717ad7e9868a6fb02/userdata/pidfile",
          "Name": "stresstest-stresstest-1",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "system_u:object_r:container_file_t:s0:c316,c844",
          "ProcessLabel": "system_u:system_r:container_t:s0:c316,c844",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/lib/containers/storage/overlay/6d7221a90c55f219a28198b75673c57c9c37043c4765147c89bdfa6858f844d1/diff:/var/lib/containers/storage/overlay/77e27c294a77bf4b1a120304beb1dce464513c1e22735acec68a7d6c59f343c9/diff:/var/lib/containers/storage/overlay/b0f72ec3d47996f5aa3780ed04587334151be03d0dbb55c413cc597fff5f3000/diff:/var/lib/containers/storage/overlay/54ebe549e39377bb9e7b9e4ab4018883539fe9d977a13540e8d0aa3f5ecbcf9a/diff:/var/lib/containers/storage/overlay/a6be224c81889aaf6fd5db9cca019b0a8222d79dc2ac19b42d6c65d196f5c640/diff:/var/lib/containers/storage/overlay/d781db8cbf993553055e6cf90365b7c079ff2e44de2142f9d682c2dfdbea0ac5/diff:/var/lib/containers/storage/overlay/65080043635c9d39784c60f905e2e276f3d44c868934a5c40778e45f20fb95ee/diff:/var/lib/containers/storage/overlay/94f3361a85151e4684928b457d917ea6b383d99c6f3afc79f8fc8b913c12446e/diff:/var/lib/containers/storage/overlay/4586c4b462dfdaa8391f74ca6960917574806d9d8bd524fea5962eb5ceeff479/diff:/var/lib/containers/storage/overlay/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef/diff",
                    "MergedDir": "/var/lib/containers/storage/overlay/69a458adeed462b0c9f77963a346933f2f9fb7fb75f4bbfd25df349ae631a908/merged",
                    "UpperDir": "/var/lib/containers/storage/overlay/69a458adeed462b0c9f77963a346933f2f9fb7fb75f4bbfd25df349ae631a908/diff",
                    "WorkDir": "/var/lib/containers/storage/overlay/69a458adeed462b0c9f77963a346933f2f9fb7fb75f4bbfd25df349ae631a908/work"
               }
          },
          "Mounts": [],
          "Dependencies": [
               "25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b"
          ],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {},
               "SandboxKey": "/run/netns/netns-d07c0562-00dd-ae74-eb0b-43d122f1fb29",
               "Networks": {
                    "podman-default-kube-network": {
                         "EndpointID": "",
                         "Gateway": "10.89.0.1",
                         "IPAddress": "10.89.0.2",
                         "IPPrefixLen": 24,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "5e:b0:af:a5:6a:dc",
                         "NetworkID": "podman-default-kube-network",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "stresstest-1",
                              "25daf95aa8e7",
                              "stresstest"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 3,
          "Config": {
               "Hostname": "stresstest",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "container=podman",
                    "HOME=/",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "HOSTNAME=stresstest"
               ],
               "Cmd": [
                    "--vm",
                    "8",
                    "--vm-bytes",
                    "5120M"
               ],
               "Image": "docker.io/progrium/stress:latest",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": "/usr/bin/stress --verbose",
               "OnBuild": null,
               "Labels": {
                    "PODMAN_SYSTEMD_UNIT": "stress.service"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.kubernetes.cri-o.SandboxID": "25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b",
                    "org.opencontainers.image.stopSignal": "15"
               },
               "StopSignal": 15,
               "HealthcheckOnFailureAction": "none",
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "sdNotifyMode": "ignore"
          },
          "HostConfig": {
               "Binds": [],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "container:25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "container:25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "container:25daf95aa8e70d94d2d9903f18c135d0a962f2ab242f26ccce587337d42e709b",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 1000000000,
               "NanoCpus": 0,
               "CgroupParent": "machine.slice/machine-libpod_pod_881e17054a4d9459300178c0893d5ca44e2e5853efa04f021328f0a6832c8571.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": -1,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 4194304,
                         "Hard": 4194304
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

Describe the results you expected

A container that was killed due to OOM can be killed automatically because its processes are not assigned an oom_score_adj of -1000, unless specifically specified.

crun version

crun --version
crun version 1.11.2
commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 65953271fc1e506ac4eb890c645f3f75976973b4'
  cpuUtilization:
    idlePercent: 97.61
    systemPercent: 1.21
    userPercent: 1.18
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: centos
    version: "9"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-285.el9.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 32724975616
  memTotal: 33390977024
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-3.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.2.2-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 2199908352
  swapTotal: 2199908352
  uptime: 0h 3m 54.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 66486038528
  graphRootUsed: 1482665984
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1699861154
  BuiltTime: Mon Nov 13 07:39:14 2023
  GitCommit: ""
  GoVersion: go1.21.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

Additional environment details

  • Logs and information above: Captured on CentOS 9, podman version 4.7.2
  • Initially observed on RHEL8, rolling release container-tools module, podman version 4.6.1, crun version 1.8.7
  • Also reproduced on CentOS 8, podman version 4.6.1
@giuseppe
Copy link
Member

crun itself doesn't handle the restart, so the bug could be in Podman. Could you please check the /run/crun/$CONTAINER_ID/config.json file for what oomScoreAdj value is specified there?

@hanneshauer
Copy link
Author

hanneshauer commented Nov 23, 2023 •
edited
Loading

There is no oomScoreAdj specified:

$ sudo cat /run/crun/c46b038330881f30ae346b1ef089d3e9c2d0002f718ea632a5228e3c1f00be0d/config.json
{
  "ociVersion": "1.1.0+dev",
  "process": {
    "user": {
      "uid": 0,
      "gid": 0,
      "umask": 18
    },
    "args": [
      "/usr/bin/stress",
      "--verbose",
      "--vm",
      "8",
      "--vm-bytes",
      "5120M"
    ],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "container=podman",
      "HOME=/",
      "HOSTNAME=stresstest"
    ],
    "cwd": "/",
    "capabilities": {
      "bounding": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE",
        "CAP_SETFCAP",
        "CAP_SETGID",
        "CAP_SETPCAP",
        "CAP_SETUID",
        "CAP_SYS_CHROOT"
      ],
      "effective": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE",
        "CAP_SETFCAP",
        "CAP_SETGID",
        "CAP_SETPCAP",
        "CAP_SETUID",
        "CAP_SYS_CHROOT"
      ],
      "permitted": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE",
        "CAP_SETFCAP",
        "CAP_SETGID",
        "CAP_SETPCAP",
        "CAP_SETUID",
        "CAP_SYS_CHROOT"
      ]
    },
    "rlimits": [
      {
        "type": "RLIMIT_NPROC",
        "hard": 4194304,
        "soft": 4194304
      }
    ],
    "selinuxLabel": "system_u:system_r:container_t:s0:c255,c478"
  },
  "root": {
    "path": "/var/lib/containers/storage/overlay/4bff9d5187240885e5eddc4d2dffae38fbffac69ec87be3c61699225d4d6f981/merged"
  },
  "mounts": [
    {
      "destination": "/tmp",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "rw",
        "rprivate",
        "nosuid",
        "nodev",
        "tmpcopyup"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/run",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "rw",
        "rprivate",
        "nosuid",
        "nodev",
        "tmpcopyup"
      ]
    },
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/var/tmp",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "rw",
        "rprivate",
        "nosuid",
        "nodev",
        "tmpcopyup"
      ]
    },
    {
      "destination": "/run/.containerenv",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/c46b038330881f30ae346b1ef089d3e9c2d0002f718ea632a5228e3c1f00be0d/userdata/.containerenv",
      "options": [
        "bind",
        "rprivate"
      ]
    },
    {
      "destination": "/run/secrets",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/c46b038330881f30ae346b1ef089d3e9c2d0002f718ea632a5228e3c1f00be0d/userdata/run/secrets",
      "options": [
        "bind",
        "rprivate"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "bind",
      "source": "/var/lib/containers/storage/overlay-containers/73210cdb136e31f7ff57b43ff6f7e6642080779f85f26fd5d177d891a853f255/userdata/shm",
      "options": [
        "bind",
        "rprivate",
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/etc/hostname",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/c46b038330881f30ae346b1ef089d3e9c2d0002f718ea632a5228e3c1f00be0d/userdata/hostname",
      "options": [
        "bind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/hosts",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/73210cdb136e31f7ff57b43ff6f7e6642080779f85f26fd5d177d891a853f255/userdata/hosts",
      "options": [
        "bind",
        "rprivate"
      ]
    },
    {
      "destination": "/etc/resolv.conf",
      "type": "bind",
      "source": "/run/containers/storage/overlay-containers/73210cdb136e31f7ff57b43ff6f7e6642080779f85f26fd5d177d891a853f255/userdata/resolv.conf",
      "options": [
        "bind",
        "rprivate"
      ]
    },
    {
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "source": "cgroup",
      "options": [
        "rprivate",
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "ro"
      ]
    }
  ],
  "annotations": {
    "io.container.manager": "libpod",
    "io.kubernetes.cri-o.SandboxID": "73210cdb136e31f7ff57b43ff6f7e6642080779f85f26fd5d177d891a853f255",
    "org.opencontainers.image.stopSignal": "15"
  },
  "linux": {
    "resources": {
      "devices": [
        {
          "allow": false,
          "access": "rwm"
        }
      ],
      "memory": {
        "limit": 1000000000
      },
      "pids": {
        "limit": 2048
      }
    },
    "cgroupsPath": "machine-libpod_pod_ef0bd5f561d6d6c84a633fb72298047ead62895e19575a81dc138547766548bb.slice:libpod:c46b038330881f30ae346b1ef089d3e9c2d0002f718ea632a5228e3c1f00be0d",
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "network",
        "path": "/proc/4448/ns/net"
      },
      {
        "type": "ipc",
        "path": "/proc/4448/ns/ipc"
      },
      {
        "type": "uts",
        "path": "/proc/4448/ns/uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "cgroup"
      }
    ],
    "seccomp": {
      "defaultAction": "SCMP_ACT_ERRNO",
      "defaultErrnoRet": 38,
      "architectures": [
        "SCMP_ARCH_X86_64",
        "SCMP_ARCH_X86",
        "SCMP_ARCH_X32"
      ],
      "syscalls": [
        {
          "names": [
            "bdflush",
            "io_pgetevents",
            "kexec_file_load",
            "kexec_load",
            "migrate_pages",
            "move_pages",
            "nfsservctl",
            "nice",
            "oldfstat",
            "oldlstat",
            "oldolduname",
            "oldstat",
            "olduname",
            "pciconfig_iobase",
            "pciconfig_read",
            "pciconfig_write",
            "sgetmask",
            "ssetmask",
            "swapcontext",
            "swapoff",
            "swapon",
            "sysfs",
            "uselib",
            "userfaultfd",
            "ustat",
            "vm86",
            "vm86old",
            "vmsplice"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "_llseek",
            "_newselect",
            "accept",
            "accept4",
            "access",
            "adjtimex",
            "alarm",
            "bind",
            "brk",
            "capget",
            "capset",
            "chdir",
            "chmod",
            "chown",
            "chown32",
            "clock_adjtime",
            "clock_adjtime64",
            "clock_getres",
            "clock_getres_time64",
            "clock_gettime",
            "clock_gettime64",
            "clock_nanosleep",
            "clock_nanosleep_time64",
            "clone",
            "clone3",
            "close",
            "close_range",
            "connect",
            "copy_file_range",
            "creat",
            "dup",
            "dup2",
            "dup3",
            "epoll_create",
            "epoll_create1",
            "epoll_ctl",
            "epoll_ctl_old",
            "epoll_pwait",
            "epoll_pwait2",
            "epoll_wait",
            "epoll_wait_old",
            "eventfd",
            "eventfd2",
            "execve",
            "execveat",
            "exit",
            "exit_group",
            "faccessat",
            "faccessat2",
            "fadvise64",
            "fadvise64_64",
            "fallocate",
            "fanotify_mark",
            "fchdir",
            "fchmod",
            "fchmodat",
            "fchown",
            "fchown32",
            "fchownat",
            "fcntl",
            "fcntl64",
            "fdatasync",
            "fgetxattr",
            "flistxattr",
            "flock",
            "fork",
            "fremovexattr",
            "fsconfig",
            "fsetxattr",
            "fsmount",
            "fsopen",
            "fspick",
            "fstat",
            "fstat64",
            "fstatat64",
            "fstatfs",
            "fstatfs64",
            "fsync",
            "ftruncate",
            "ftruncate64",
            "futex",
            "futex_time64",
            "futimesat",
            "get_mempolicy",
            "get_robust_list",
            "get_thread_area",
            "getcpu",
            "getcwd",
            "getdents",
            "getdents64",
            "getegid",
            "getegid32",
            "geteuid",
            "geteuid32",
            "getgid",
            "getgid32",
            "getgroups",
            "getgroups32",
            "getitimer",
            "getpeername",
            "getpgid",
            "getpgrp",
            "getpid",
            "getppid",
            "getpriority",
            "getrandom",
            "getresgid",
            "getresgid32",
            "getresuid",
            "getresuid32",
            "getrlimit",
            "getrusage",
            "getsid",
            "getsockname",
            "getsockopt",
            "gettid",
            "gettimeofday",
            "getuid",
            "getuid32",
            "getxattr",
            "inotify_add_watch",
            "inotify_init",
            "inotify_init1",
            "inotify_rm_watch",
            "io_cancel",
            "io_destroy",
            "io_getevents",
            "io_setup",
            "io_submit",
            "ioctl",
            "ioprio_get",
            "ioprio_set",
            "ipc",
            "keyctl",
            "kill",
            "landlock_add_rule",
            "landlock_create_ruleset",
            "landlock_restrict_self",
            "lchown",
            "lchown32",
            "lgetxattr",
            "link",
            "linkat",
            "listen",
            "listxattr",
            "llistxattr",
            "lremovexattr",
            "lseek",
            "lsetxattr",
            "lstat",
            "lstat64",
            "madvise",
            "mbind",
            "membarrier",
            "memfd_create",
            "memfd_secret",
            "mincore",
            "mkdir",
            "mkdirat",
            "mknod",
            "mknodat",
            "mlock",
            "mlock2",
            "mlockall",
            "mmap",
            "mmap2",
            "mount",
            "mount_setattr",
            "move_mount",
            "mprotect",
            "mq_getsetattr",
            "mq_notify",
            "mq_open",
            "mq_timedreceive",
            "mq_timedreceive_time64",
            "mq_timedsend",
            "mq_timedsend_time64",
            "mq_unlink",
            "mremap",
            "msgctl",
            "msgget",
            "msgrcv",
            "msgsnd",
            "msync",
            "munlock",
            "munlockall",
            "munmap",
            "name_to_handle_at",
            "nanosleep",
            "newfstatat",
            "open",
            "open_tree",
            "openat",
            "openat2",
            "pause",
            "pidfd_getfd",
            "pidfd_open",
            "pidfd_send_signal",
            "pipe",
            "pipe2",
            "pivot_root",
            "pkey_alloc",
            "pkey_free",
            "pkey_mprotect",
            "poll",
            "ppoll",
            "ppoll_time64",
            "prctl",
            "pread64",
            "preadv",
            "preadv2",
            "prlimit64",
            "process_mrelease",
            "process_vm_readv",
            "process_vm_writev",
            "pselect6",
            "pselect6_time64",
            "ptrace",
            "pwrite64",
            "pwritev",
            "pwritev2",
            "read",
            "readahead",
            "readdir",
            "readlink",
            "readlinkat",
            "readv",
            "reboot",
            "recv",
            "recvfrom",
            "recvmmsg",
            "recvmmsg_time64",
            "recvmsg",
            "remap_file_pages",
            "removexattr",
            "rename",
            "renameat",
            "renameat2",
            "restart_syscall",
            "rmdir",
            "rseq",
            "rt_sigaction",
            "rt_sigpending",
            "rt_sigprocmask",
            "rt_sigqueueinfo",
            "rt_sigreturn",
            "rt_sigsuspend",
            "rt_sigtimedwait",
            "rt_sigtimedwait_time64",
            "rt_tgsigqueueinfo",
            "sched_get_priority_max",
            "sched_get_priority_min",
            "sched_getaffinity",
            "sched_getattr",
            "sched_getparam",
            "sched_getscheduler",
            "sched_rr_get_interval",
            "sched_rr_get_interval_time64",
            "sched_setaffinity",
            "sched_setattr",
            "sched_setparam",
            "sched_setscheduler",
            "sched_yield",
            "seccomp",
            "select",
            "semctl",
            "semget",
            "semop",
            "semtimedop",
            "semtimedop_time64",
            "send",
            "sendfile",
            "sendfile64",
            "sendmmsg",
            "sendmsg",
            "sendto",
            "set_mempolicy",
            "set_robust_list",
            "set_thread_area",
            "set_tid_address",
            "setfsgid",
            "setfsgid32",
            "setfsuid",
            "setfsuid32",
            "setgid",
            "setgid32",
            "setgroups",
            "setgroups32",
            "setitimer",
            "setns",
            "setpgid",
            "setpriority",
            "setregid",
            "setregid32",
            "setresgid",
            "setresgid32",
            "setresuid",
            "setresuid32",
            "setreuid",
            "setreuid32",
            "setrlimit",
            "setsid",
            "setsockopt",
            "setuid",
            "setuid32",
            "setxattr",
            "shmat",
            "shmctl",
            "shmdt",
            "shmget",
            "shutdown",
            "sigaction",
            "sigaltstack",
            "signal",
            "signalfd",
            "signalfd4",
            "sigpending",
            "sigprocmask",
            "sigreturn",
            "sigsuspend",
            "socket",
            "socketcall",
            "socketpair",
            "splice",
            "stat",
            "stat64",
            "statfs",
            "statfs64",
            "statx",
            "symlink",
            "symlinkat",
            "sync",
            "sync_file_range",
            "syncfs",
            "syscall",
            "sysinfo",
            "syslog",
            "tee",
            "tgkill",
            "time",
            "timer_create",
            "timer_delete",
            "timer_getoverrun",
            "timer_gettime",
            "timer_gettime64",
            "timer_settime",
            "timer_settime64",
            "timerfd",
            "timerfd_create",
            "timerfd_gettime",
            "timerfd_gettime64",
            "timerfd_settime",
            "timerfd_settime64",
            "times",
            "tkill",
            "truncate",
            "truncate64",
            "ugetrlimit",
            "umask",
            "umount",
            "umount2",
            "uname",
            "unlink",
            "unlinkat",
            "unshare",
            "utime",
            "utimensat",
            "utimensat_time64",
            "utimes",
            "vfork",
            "wait4",
            "waitid",
            "waitpid",
            "write",
            "writev"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 0,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 8,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 131072,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 131080,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "personality"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 4294967295,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "arch_prctl"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "modify_ldt"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "open_by_handle_at"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "bpf",
            "fanotify_init",
            "lookup_dcookie",
            "perf_event_open",
            "quotactl",
            "setdomainname",
            "sethostname",
            "setns"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "chroot"
          ],
          "action": "SCMP_ACT_ALLOW"
        },
        {
          "names": [
            "delete_module",
            "finit_module",
            "init_module",
            "query_module"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "acct"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "kcmp",
            "process_madvise"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "ioperm",
            "iopl"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "clock_settime",
            "clock_settime64",
            "settimeofday",
            "stime"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "vhangup"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 1
        },
        {
          "names": [
            "socket"
          ],
          "action": "SCMP_ACT_ERRNO",
          "errnoRet": 22,
          "args": [
            {
              "index": 0,
              "value": 16,
              "op": "SCMP_CMP_EQ"
            },
            {
              "index": 2,
              "value": 9,
              "op": "SCMP_CMP_EQ"
            }
          ]
        },
        {
          "names": [
            "socket"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 2,
              "value": 9,
              "op": "SCMP_CMP_NE"
            }
          ]
        },
        {
          "names": [
            "socket"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 0,
              "value": 16,
              "op": "SCMP_CMP_NE"
            }
          ]
        },
        {
          "names": [
            "socket"
          ],
          "action": "SCMP_ACT_ALLOW",
          "args": [
            {
              "index": 2,
              "value": 9,
              "op": "SCMP_CMP_NE"
            }
          ]
        }
      ]
    },
    "maskedPaths": [
      "/proc/acpi",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/proc/scsi",
      "/sys/firmware",
      "/sys/fs/selinux",
      "/sys/dev/block",
      "/sys/devices/virtual/powercap"
    ],
    "readonlyPaths": [
      "/proc/asound",
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ],
    "mountLabel": "system_u:object_r:container_file_t:s0:c255,c478"
  }
}

@giuseppe
Copy link
Member

when the oomscore is not specified, crun doesn't change it.

This seems like a Podman issue, thanks for the additional information.

@giuseppe giuseppe transferred this issue from containers/crun Nov 23, 2023
@giuseppe
Copy link
Member

the regression was introduced by conmon containers/conmon@813c8d7

giuseppe added a commit to giuseppe/conmon that referenced this issue Nov 23, 2023
@giuseppe
oom-score: restore oom score before running exit command
2deff03 
When the exit command is executed, the oom score is restored to its
original value otherwise the command runs with -1000.

Closes: containers/podman#20765

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Nov 23, 2023
Merged
@giuseppe
Copy link
Member

proposed patch: containers/conmon#464

@hanneshauer
Copy link
Author

the regression was introduced by conmon containers/conmon@813c8d7

I can confirm that I can not see this problem after downgrading to conmon v2.1.7 in my testing scenarios

giuseppe added a commit to giuseppe/conmon that referenced this issue Nov 28, 2023
@giuseppe
oom-score: restore oom score before running exit command
ba335ee 
When the exit command is executed, the oom score is restored to its
original value otherwise the command runs with -1000.

Closes: containers/podman#20765

Signed-off-by: Giuseppe Scrivano <[email protected]>
haircommander pushed a commit to containers/conmon that referenced this issue Nov 28, 2023
@giuseppe @haircommander
oom-score: restore oom score before running exit command
633ac6b 
When the exit command is executed, the oom score is restored to its
original value otherwise the command runs with -1000.

Closes: containers/podman#20765

Signed-off-by: Giuseppe Scrivano <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Feb 27, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

oom-score: restore oom score before running exit command giuseppe/conmon
2 participants
@giuseppe @hanneshauer