Compactor OOM

[acarsercan]

Hi,

• Thanos v0.8.1
• banzai thanos helm chart - https://github.com/banzaicloud/banzai-charts/blob/master/thanos/templates/compact-deployment.yaml
• S3 bucket 200GB

I been using thanos behind a prometheus-operator deployment for over a month. During holiday period (out of hours) memory usage has skyrocketed. I'm still unable to pin down what is causing it.

Compactor pod will OOM every 15-20 minutes, going beyond the 8GB limit

I'm going to guess there is a setting or retention which I just hit.

Any ideas?

Log level debug

 level=debug ts=2019-12-27T16:32:34.090393606Z caller=main.go:101 msg="maxprocs: Updating GOMAXPROCS=[2]: determined from CPU quota"
 level=info ts=2019-12-27T16:32:34.090586411Z caller=main.go:149 msg="Tracing will be disabled"
 level=info ts=2019-12-27T16:32:34.090762472Z caller=factory.go:43 msg="loading bucket configuration"
 level=info ts=2019-12-27T16:32:34.105826606Z caller=compact.go:270 msg="retention policy of raw samples is enabled" duration=720h0m0s
 level=info ts=2019-12-27T16:32:34.105858745Z caller=compact.go:273 msg="retention policy of 5 min aggregated samples is enabled" duration=2880h0m0s
 level=info ts=2019-12-27T16:32:34.105869173Z caller=compact.go:276 msg="retention policy of 1 hour aggregated samples is enabled" duration=8760h0m0s
 level=info ts=2019-12-27T16:32:34.105882534Z caller=compact.go:361 msg="starting compact node"
 level=info ts=2019-12-27T16:32:34.105895111Z caller=prober.go:107 msg="changing probe status" status=ready
 level=info ts=2019-12-27T16:32:34.105967171Z caller=prober.go:127 msg="changing probe status" status=healthy
 level=info ts=2019-12-27T16:32:34.10600956Z caller=http.go:53 service=http/server component=compact msg="listening for requests and metrics" address=0.0.0.0:10902
 level=info ts=2019-12-27T16:32:34.109593018Z caller=compact.go:1083 msg="start sync of metas"
 level=debug ts=2019-12-27T16:32:34.175253071Z caller=compact.go:288 msg="download meta" block=01DTR7TX8GGMXM1Z3WQ2JKHC3J
 level=debug ts=2019-12-27T16:32:34.175278297Z caller=compact.go:288 msg="download meta" block=01DTR7TND1QJEH1JCTWKPDQR8M
...

Regards

[GiedriusS]

Hi, could you upload the heap profile of Thanos Compact when this starts happening? Perhaps something will be there. Also, we could definitely provide some guidance on how much RAM is needed. Nonetheless, most likely this is expected - my guess would be that it was downloading the files before starting the compaction/downsampling process.

[acarsercan]

I sure will provide heap profile when this happens again. Any ideas what might of triggered this? (ie a setting? ) We were happily running thanos for few months.

Regards