[SPIKE] Research why Epiphany nodes hang when memory is overcommited

[sk4zuzu]

Is your feature request related to a problem? Please describe.
It seems that when k8s resource limits are incorrectly defined and memory is overcommited by applications user loses all ssh access to Epiphany VMs. This indicates that operating system is not protected from memory and cpu overusage. Usually it's not possible to debug such machines, restart is required and cluster becomes unstable. Since OS is not protected there is also security risk of possible denial-of-service attacks. 😱

I suppose it's likely systemd slices are not correctly configured on Epiphany VMs in 0.8 release (at least).

Describe the solution you'd like
What we need to do in this spike is decide what is the culprit here and research ways to prevent such situations in the future. We need confirmation also that this situation happens.

Describe alternatives you've considered
None.

Additional context
Logs found on affected nodes:

Killed process 13428 (java) total-vm:14218144kB, anon-rss:2096560kB, file-rss:3684kB, shmem-rss:0kB, UID:0 pgtables:25228kB oom_score_adj:998

kernel: [200403.323264] memory: usage 3145696kB, limit 3145728kB, failcnt 17191

dockerd[14005]: time="2020-11-25T21:40:40.609327299Z" level=warning msg="Your kernel does not support swap limit capabilities,or the cgroup is not mounted. Memory limited without swap."

 kernel: [ 0.289359] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]

[sk4zuzu]

I was basically unable to reproduce the issue in a generic way. I tried Ubuntu and JVM versions exactly as on the faulty environment:

  storage_image_reference:
    publisher: Canonical
    offer: UbuntuServer
    sku: 18.04-LTS
    version: 18.04.202006101

openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment (build 11.0.5+10-alpine-r0)
OpenJDK 64-Bit Server VM (build 11.0.5+10-alpine-r0, mixed mode)

Epiphany's Kubernetes behaved well in every attempt to crash node VMs. I think the next step would be to work closely with the team that experienced the issue and carefuly observe what their software does. 🤔

@to-bar thank you for very nice memory stress testing toolset! 👍😍

[mkyc]

@sk4zuzu what is next step here in this task? Why is it blocked?

[sk4zuzu]

I got vpn access, I'm looking at the cluster where it all happened, because "generic" tests showed no issues. I can move it to in progress now I think and continue examining the cluster.

[przemyslavic]

Tested
✔ epicli apply
✔ epicli upgrade from 0.6, 0.7 HA, 0.8 to develop
After patching cgroup driver (switching from cgroupfs to systemd) it looks to be working properly.