docker oom-killer / possible memory leak ?

[jdel]

Hello,

I have been using dockbeat for over 2 months to gather container stats from my Rancher managed infrastructure.

I am using 1.0.0 release of dockbeat, in an alpine:3.4 with glibc container and all of my servers running the dockbeat container have logs similar to this:

Nov 25 13:34:49 rancher kernel: [4159464.630005] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 26 08:50:23 rancher kernel: [4228783.775652] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 27 04:36:56 rancher kernel: [4299961.356662] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 27 23:53:57 rancher kernel: [4369367.512043] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 28 19:11:31 rancher kernel: [4438805.604295] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 29 14:28:33 rancher kernel: [4508213.493451] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Nov 30 09:45:06 rancher kernel: [4577591.614083] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  1 05:28:39 rancher kernel: [4648589.332368] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  2 01:19:43 rancher kernel: [4720036.981089] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  2 21:07:16 rancher kernel: [4791274.704217] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  3 16:22:49 rancher kernel: [4860592.882285] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  4 12:08:53 rancher kernel: [4931741.679099] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  5 07:22:56 rancher kernel: [5000969.824923] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  6 02:43:29 rancher kernel: [5070587.938654] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  6 21:48:32 rancher kernel: [5139276.202580] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  7 16:38:34 rancher kernel: [5207063.934761] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  8 11:28:38 rancher kernel: [5274852.423997] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec  9 07:03:11 rancher kernel: [5345310.312589] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 10 02:13:14 rancher kernel: [5414298.531009] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 10 21:04:47 rancher kernel: [5482176.791567] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 11 15:58:49 rancher kernel: [5550205.009526] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 12 11:07:23 rancher kernel: [5619103.233274] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 13 05:52:56 rancher kernel: [5686621.833695] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 14 01:06:57 rancher kernel: [5755848.022659] dockbeat invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0

My dockbeat container is currently limited to using 100MB of ram and 200MB of swap (mem_limit set in docker compose).

What is the expected memory usage of dockbeat ?
Could it be possible that there is a memory leak somewhere ?

[luckydonald]

It really looks like.

[ghost]

Hi,
Same issue in our side. Is anything foreseen to fix this memory leak?

[ruflin]

It would be interesting to hear if you have the same issues with the docker module in metricbeat.

[jdel]

Any news on this ? I am seriously considering using something else...

[douaejeouit]

@jdel We have migrated Dockbeat to a new docker module in Metricbeat. This last is a part of the Elastic stack. You can take advantage of the power of Elasricsearch and the great visualization of Kibana for a better monitoring of your Docker infrastructure. All that you need will be generated automatically, so you don't have to worry about the index template anymore, it will be loaded automatically and will be used if new indices are created. Thus you will be able to ship your data easily to elasticsearch!
To monitor your servers in Kibana, sample dashboards are provided as examples ( You can obviously customize them to feet your particular needs, it's even recommended). You can then import them into Kibana and enjoy your monitoring.
If you need any further information, don't hesitate to contact us. I will be pleased to answer your questions. And please let me know if you still have the same issues using the docker module!
Thank you.
https://www.elastic.co/products/beats/metricbeat
https://www.elastic.co/blog/beats-5-1-1-released
https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-overview.html

[douaejeouit]

@jdel The issue was found & resolved in Metricbeat thanks to @ruflin #3514

[luckydonald]

This looks promising