Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

-m, unexpected behavior? #97

Closed
hakavlad opened this issue Nov 1, 2018 · 33 comments
Closed

-m, unexpected behavior? #97

hakavlad opened this issue Nov 1, 2018 · 33 comments

Comments

@hakavlad
Copy link
Contributor

hakavlad commented Nov 1, 2018 •
edited
Loading 

  -m PERCENT[,KILL_PERCENT] set available memory minimum to PERCENT of total (default 10 %).
                            earlyoom sends SIGTERM once below PERCENT, then SIGKILL once below
                            KILL_PERCENT (default PERCENT/2).

I do not understand how this option works.

earlyoom -m 22[,20]
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 22 %, SIGKILL at 11 %
swap total: 0 MiB, sending SIGTERM at 10 %, SIGKILL at 5 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 3697 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3693 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)

Is it normal?

How to set 22% for SIGTERM and 20% for SIGKILL?
Explain this in more detail, please.

Another example:

earlyoom -m 22[,16] -s 18[,28]
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 22 %, SIGKILL at 11 %
swap total: 0 MiB, sending SIGTERM at 18 %, SIGKILL at 9 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 3659 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)
@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

earlyoom -M 220[,160] -S 180[,170]
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '180[,170]' exceeds limit 0
WTF?

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

Remove the "[ ]", for example,

earlyoom -m 22,20
earlyoom v1.1-54-g20e4a01-dirty
mem  total: 7835 MiB, sending SIGTERM at 22 %, SIGKILL at 20 %
swap total:    0 MiB, sending SIGTERM at 10 %, SIGKILL at  5 %

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

The "[ ]" just mean that the value is optional. I will add examples to make that clear

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

Ok, thank you.

What about

earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '180000,170000' exceeds limit 0

?

rfjakob added a commit that referenced this issue Nov 1, 2018
@rfjakob
MANPAGE: add example for KILL_PERCENT
23228fb 
The new optional KILL_PERCENT is confusing to users,
add examples.

#97
@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '200000,100000' exceeds limit 0

It is not normal.

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

Strange. How much swap do you have? (output of free -k ?)

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

swap = 0 MiB

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

Well, if you have 0 swap, you cannot set -S to anything above zero.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

with swap works normal

earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 10 %, SIGKILL at 5 %
swap total: 5875 MiB, sending SIGTERM at 3 %, SIGKILL at 1 %
mem avail: 3621 of 5875 MiB (61 %), swap free: 5875 of 5875 MiB (100 %)
mem avail: 3618 of 5875 MiB (61 %), swap free: 5875 of 5875 MiB (100 %)

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

cannot set -S to anything above zero

Maybe It is not well.

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

Hmm, earlyoom v1.1 had a better message, but it would still exit with an error:

earlyoom -S 1000
earlyoom v1.1-26-g867d0e8-dirty
-S: the value you passed (1000 kiB) is above total swap (0 kiB)

The error message is different in v1.2 because the parsing of the "X,Y" values is more complicated. I'll try to improve it.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

I can start earlyoom with swap:
earlyoom -M 220000,160000 -S 180000,170000
and all is well. Next, I turn off the swap and all is well again, earlyoom works without swap with options earlyoom -M 220000,160000 -S 180000,170000

But I can not start earlyoom without swap with this options. Why?

Next, If I set earlyoom with -M 220000,160000 -S 180000,170000 In /etc, it will works fine. Next I will disable swap and restart earlyoom: Here the daemon will fall!

@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

Hmm, should earlyoom ignore the swap settings when there is zero swap? What do you think?

PS: If you restart with less RAM, the daemon will fail as well

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

should earlyoom ignore the swap settings when there is zero swap?

Only if SwapTotal = 0.
It seems obvious to me that swap settings should only be applied when SwapTotal>0.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

@rfjakob Do you agree with me?

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

How it can works if swap disabled.

Config:

    MemAvailable levels.

mem_min_sigterm = 10 %
mem_min_sigkill = 5 %

    SwapFree levels.

swap_min_sigterm = 200M
swap_min_sigkill = 100M

Output:

sudo nohang
The path to the config: /etc/nohang/nohang.conf
Monitoring started!
TRIGGERED!
  MemAvailable (577 MiB, 9.8 %) < mem_min_sigterm (588 MiB, 10.0 %)
  SwapFree (0 MiB, 0.0 %) < swap_min_sigterm (200 MiB, - %)
  Finding the process with the highest badness
  Victim is python3, pid: 13654, badness: 0, VmRSS: 4141 MiB, VmSwap: 0 MiB
  Sending SIGTERM to the victim; signal received; response time: 18 ms

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '200000,100000' exceeds limit 0

It is not normal, it should be fixed, IMHO.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

mem avail: 1110 of 5875 MiB (18 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 989 of 5875 MiB (16 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 867 of 5875 MiB (14 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 744 of 5875 MiB (12 %), swap free: 0 of 0 MiB ( 0 %)
Low Memory! At or below sigterm limits (mem: 10 %, swap: 3 %)
mem avail: 635 of 5875 MiB (10 %), swap free: 0 of 0 MiB ( 0 %)
Killing process 'python3' with signal 15, pid: 14069, badness: 583, VmRSS: 3422 MiB
mem avail: 2336 of 5875 MiB (39 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 4059 of 5875 MiB (69 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 4059 of 5875 MiB (69 %), swap free: 0 of 0 MiB ( 0 %)

Earlyoom can works with -S 200000,100000 if swap will be disabled after earlyoom starts.

Why it is forbidden to start with -S 200000,100000 without swap?

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

How about this

  -m SIGTERM_PERCENT,SIGKILL_PERCENT set available memory minimum to SIGTERM_PERCENT of total (default 10 %).
                                     Second value is optional and by default SIGKILL_PERCENT = 1/2 SIGTERM_PERCENT
                                     earlyoom sends SIGTERM once below SIGTERM_PERCENT, then SIGKILL once below SIGKILL_PERCENT.
  -s SIGTERM_PERCENT,SIGKILL_PERCENT set free swap minimum to SIGTERM_PERCENT of total (default 10 %),
                                     second value is optional and by default equals 1/2 SIGTERM_PERCENT
  -M SIGTERM_SIZE,SIGKILL_SIZE       set available memory minimum to SIGTERM_SIZE KiB (second value is optional)
  -S SIGTERM_SIZE,SIGKILL_SIZE       set free swap minimum to SIGTERM_SIZE KiB (second value is optional)

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

or this

  -m SIGTERM_PERCENT,SIGKILL_PERCENT set available memory minimum to SIGTERM_PERCENT of total (default 10 %).
                                     Second value is optional and by default SIGKILL_PERCENT = 1/2 SIGTERM_PERCENT
                                     earlyoom sends SIGTERM once below SIGTERM_PERCENT, then SIGKILL once below SIGKILL_PERCENT.
  -s SIGTERM_PERCENT,SIGKILL_PERCENT set free swap minimum to SIGTERM_PERCENT of total (default 10 %),
                                     second value is optional and by default equals 1/2 SIGTERM_PERCENT
  -M SIGTERM_KiB,SIGKILL_KiB         set available memory minimum to SIGTERM_KiB KiB (second value is optional)
  -S SIGTERM_KiB,SIGKILL_KiB         set free swap minimum to SIGTERM_KiB KiB (second value is optional)

?

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

earlyoom -m 5,9
earlyoom v1.2-2-gb52cc56
fatal: -m: sigkill value exceeds sigterm value in '5,9'

In this case I expect 9% for SIGKILL and ignore SIGTERM, not fatal error.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

earlyoom -m 5,0
earlyoom v1.2-2-gb52cc56
fatal: -m: zero sigkill value in '5,0'

In this case I expect 5% SIGTERM and ignore SIGKILL

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018

SIGTERM_PERCENT should be ignored if SIGKILL_PERCENT > SIGTERM_PERCENT, not daemon fails.
0 value shouldn't cause err. It should cause only ignoring the signal.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 1, 2018 •
edited
Loading

Also, the documentation does not indicate what values SIGTERM_PERCENT and SIGKILL_SIZE can take (forbidden zero value and KLL can't be higher than TERM size)

rfjakob added a commit that referenced this issue Nov 1, 2018
@rfjakob
tests: add unit tests for parse_term_kill_tuple
79f5d43 
#97
rfjakob added a commit that referenced this issue Nov 1, 2018
@rfjakob
Allow zero SIGKILL value
b1c0483 
You can get a zero SIGKILL percentage by using

  earlyoom -m 1

so why not allow it on the command line?

#97
rfjakob added a commit that referenced this issue Nov 1, 2018
@rfjakob
Allow SIGTERM value below SIGKILL
28084c3 
Saying

  earlyoom -m 0,15

disables SIGTERM completely. Allow it.

#97
@rfjakob
Copy link
Owner

rfjakob commented Nov 1, 2018

These two work now:

earlyoom -m 5,0

earlyoom -m 5,9

rfjakob added a commit that referenced this issue Nov 3, 2018
@rfjakob
Set SIGTERM = SIGKILL if SIGTERM < SIGKILL
a8f30d7 
As stated in the man page, the preferred way to disable SIGTERM
is to set SIGTERM = SIGKILL.

For a better user experience, don't exit with a fatal error is
the user sets it to zero or below SIGKILL, but fix the value.

#97
rfjakob added a commit that referenced this issue Nov 3, 2018
@rfjakob
Ignore -S if there is no swap
582ac1b 
Instead of exiting with a fatal error, ignore "-S" if there is
zero swap space.

#97
@rfjakob
Copy link
Owner

rfjakob commented Nov 3, 2018

Startup succeeds now with 0 swap even if you pass -S. It cannot calculate the percentages, so it uses defaults.

@rfjakob
Copy link
Owner

rfjakob commented Nov 3, 2018

About the help message, I like it how it is. I think the square brackets are the usual way to show that a value is optional. Thanks for the report, I think this is done!

@rfjakob rfjakob closed this as completed Nov 3, 2018
@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 4, 2018 •
edited
Loading

this is done!

Sorry, seems like No!

earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2-12-ge600459
warning: -S: total swap is zero, using default percentages
mem total: 5875 MiB, swap total: 0 MiB
Sending SIGTERM when mem <= 3 % and swap <= 10 %,
SIGKILL when mem <= 2 % and swap <= 5 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 4000 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3996 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3997 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
...
mem avail: 0 of 5875 MiB ( 0 %), swap free: 3053 of 5875 MiB (51 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 2283 of 5875 MiB (38 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 1521 of 5875 MiB (25 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 804 of 5875 MiB (13 %)
Low Memory! At or below SIGTERM limits (mem: 3 %, swap: 10 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 576 of 5875 MiB ( 9 %)
Killing process 'tail' with signal 15, pid: 5355, badness: 760, VmRSS: 4830 MiB
mem avail: 4933 of 5875 MiB (83 %), swap free: 4523 of 5875 MiB (76 %)
mem avail: 4917 of 5875 MiB (83 %), swap free: 4529 of 5875 MiB (77 %)
mem avail: 4914 of 5875 MiB (83 %), swap free: 4531 of 5875 MiB (77 %)

In this case I expect -S 180000,170000, not 10% (after enable swap)
Using 10% is also unexpected behavior. User set exact -S 180000,170000!
After mount swapspace earlyoom should use -S 180000,170000, not 10% that was set at startup.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 4, 2018 •
edited
Loading

Good practice:

earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2...
warning: -S: total swap is zero, that's why swap will be ignored now. After mounting swapspace earlyoom will use -S 18000,17000

@rfjakob
Copy link
Owner

rfjakob commented Nov 4, 2018

Internally, earlyoom always uses percentages.
The -S and -M options are there for user friendlyness, but are converted to percentages at startup. If there is no swap, we cannot calculate the percentages, and we use default values.

I think this is good enough, and i do not want to change the internal design for this.

@hakavlad
Copy link
Contributor Author

hakavlad commented Nov 4, 2018

You just don't want to fix the bug.

@rfjakob
Copy link
Owner

rfjakob commented Nov 4, 2018

Yes.

rfjakob added a commit that referenced this issue Nov 4, 2018
@rfjakob
MANPAGE: document that -S does not work with zero swap
8ff3800 
#97
@rfjakob
Copy link
Owner

rfjakob commented Nov 4, 2018

And now I have even documented it: https://github.com/rfjakob/earlyoom/blob/master/MANPAGE.md#bugs
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rfjakob @hakavlad