Tailing pod logs with higher log rate fails.

[evilbulgarian]

Description

We notices that on the flatcaros nodes when tailing pods with higher log volume tailing the logs fails. After increasing fs.inotify.max_user_watches seems to alleviate the issue.

Impact

debugging cluster issues is impacted

Environment and steps to reproduce

1. flatcaros stable nodes running busy containers
2. kubectl logs | stern logs should work on busy containers

Expected behavior

stern and logs should work even on busy containers

Additional information

[jepio]

hi @evilbulgarian,
According to kernel docs the default value of fs.inotify.max_user_watches automatically scales with available memory at boot.

What value does your node have by default, and what value is sufficient for your usecase?

[evilbulgarian]

FlatcarOS Setup:

fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 1007804
user.max_inotify_instances = 128
user.max_inotify_watches = 1007804
core@ip-10-37-4-218 ~ $ free -m
               total        used        free      shared  buff/cache   available
Mem:          127684       30225       79784         260       17674       96004
Swap:              0           0           0

Using: fs.inotify.max_user_watches: 524288