Query via Thanos causes Prometheus to OOM #455
Comments
|
Ok, this is very interesting. Especially that part for querying Prometheus directly and through thanos-query and having OOM while doing for query. So you are saying that Prometheus OOMs.. Can you isolate that setup and turn off store and all sidecars + Proms but one and try to repro? Questions:
One fact that matters here: Sidecar (so queries via Thanos-query) is using Prometheus |
|
Hi, @Bplotka,
I'm attaching SVG heap traces from PPROF tool , hoping there's something suspicious visible there. I'm also including With ThanosWithout ThanosWe even tried running the setup on a physical machine with 700GB of RAM, and we managed to to reach ~300GB of memory usage by Prometheus container when running 5-simultenious queries Again, even though it looks like the memory usage for queries via Thanos seems much bigger, maybe we just trying to overcome the speed of light here 🙂 Any guidance on how to properly size for the Thanos (and Prometheus) deployment (e.g. maybe there's a way to see how many samples Thanos-Store sees?) would help a lot as well. |
|
I don't get one thing.. Why do you think Thanos Store matters here (or anything in the bucket) if the Prometheus container OOMs not any Thanos component? |
|
That happens when query goes through Thanos-Query, and does not seem to happen when it's issued dirrectly to Prometheus. |
|
I doubt thanos store matters here. Can you disable thanos store for now? (: And try to repro without it? |
|
How much series you have in the Prometheus? So basically what can happen is because query evaluation is on Thanos Query so maybe just data size is too big to be sent between Prometheus and sidecar? but that sounds unrealistic. We might want to exactly see how remote_read request looks like. |
|
Hi @Bplotka, We we have around 8.5mil ( You mentioned, that queries via Thanos uses
Of course, in that case, that would mean it's an issue with Prometheus, not with Thanos. |
|
Plus this: #488 (: |
|
Anybody an idea what's the state on this? We currently have something which looks like the same issue. When Grafana queries prometheus directly, the dashboard finishes in just a couple of seconds. When querying through thanos query, memory usage of both sidecar and prometheus blow up, which will eventually lead to OOM. |
|
Having the same issues, queries through grafana crash an 8 GB Prometheus instance while queries through Thanos Querier is blowing away memory. Prometheus Spec apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
flux.weave.works/antecedent: base-monitoring:helmrelease/prom-operator
creationTimestamp: "2019-06-27T07:52:44Z"
generation: 15
labels:
app: prometheus-operator-prometheus
chart: prometheus-operator-5.15.0
heritage: Tiller
release: prom-operator
name: prom-operator-prometheus-o-prometheus
namespace: base-monitoring
resourceVersion: "4214847"
selfLink: /apis/monitoring.coreos.com/v1/namespaces/base-monitoring/prometheuses/prom-operator-prometheus-o-prometheus
uid: 8c33fb10-98b0-11e9-8c8c-4e7d547fa7d0
spec:
additionalScrapeConfigs:
key: additional-scrape-configs.yaml
name: prom-operator-prometheus-o-prometheus-scrape-confg
alerting:
alertmanagers:
- name: prom-operator-prometheus-o-alertmanager
namespace: base-monitoring
pathPrefix: /
port: web
baseImage: quay.io/prometheus/prometheus
enableAdminAPI: false
externalLabels:
cluster: prod001
geo: eu
region: euw
externalUrl: http://prom-operator-prometheus-o-prometheus.base-monitoring:9090
listenLocal: false
logFormat: logfmt
logLevel: info
paused: false
replicas: 2
resources:
limits:
cpu: "2"
memory: 8Gi
requests:
cpu: 100m
memory: 3Gi
retention: 30d
routePrefix: /
ruleNamespaceSelector: {}
ruleSelector:
matchLabels:
release: prom-operator
secrets:
- istio.default
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prom-operator-prometheus-o-prometheus
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
release: prom-operator
storage:
volumeClaimTemplate:
selector: {}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: default
thanos:
image: quay.io/thanos/thanos:v0.6.0
objectStorageConfig:
key: thanos.yaml
name: thanos-objstore-config
resources:
limits:
cpu: "2"
memory: 6Gi
requests:
cpu: 100m
memory: 2Gi
tag: v0.6.0
version: v2.10.0The nodes them self have sufficient resources available, and in a direct comparison, Grafana queries don't cause any load on the Prometheus instances. We have two clusters in this scenario, one with the Prometheus instance, one with Thanos Query and Grafana. Query goes only to metrics stored in Prometheus. Maybe it would be a good idea to decrease retention time to some hours instead of 30d ? There are 236.834 Timeseries in the server right now. Currently the ingestion rate is about 15k This is the query I was running over a duration of 7 days. histogram_quantile(0.50, sum(rate(istio_request_duration_seconds_bucket{cluster="prod001", destination_workload_namespace=~"mdr-.*", source_app="istio-ingressgateway", destination_workload!="carstreamingservice"}[6h])) by (le, destination_workload_namespace)) |
|
@alexdepalex , @mkjoerg |
|
Unfortunately, we're not running on k8s yet. I'll check our config tomorrow, but it seems it's more related to the issues with remote read endpoint. Anyway, I'm still waiting for a "release" version of v2.12.0-rc.0-rr-streaming in order to test it. |
|
this is fixed with the remote read streaming. Mem usage now is few times lower: Upgrade to the latest Thanos version and enjoy this great improvement 😺 |
|
Still waiting for the next prometheus release which includes streaming, right? |
|
Yep, with the new Sidecar the RAM usage is constant and Prometheus uses noticeably less RAM due to how the new remote read interface works. @krasi-georgiev thanks for bumping this ticket! Closing this. |
|
@alexdepalex aah yeah the PR got merged just after 2.12 was cut so yeah need to wait for 2.13 or just use the master image. |
|
Wait or use our image we prepared Which is essentially 2.12 + remote read
extended protocol: `quay.io/thanos/prometheus:v2.12.0-rc.0-rr-streaming`
It's used in production already. (:
…On Fri, 13 Sep 2019 at 11:02, Krasi Georgiev ***@***.***> wrote:
@alexdepalex <https://github.com/alexdepalex> aah yeah the PR got merged
just after 2.12 was cut so yeah need to wait for 2.13 or just use the
master image.
https://github.com/prometheus/prometheus/commits/master?after=26e8d25e0b0d3459e5901805de992acf1d5eeeaa+34
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#455?email_source=notifications&email_token=ABVA3O4KYHM24RFE7NE6VQ3QJNQJ7A5CNFSM4FNHZQ32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6USLPI#issuecomment-531178941>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABVA3O7O4UXW6DV4H3JCNZLQJNQJ7ANCNFSM4FNHZQ3Q>
.
|
|
"High load on all prometheus nodes due to query fanout from thanos querier" We have implemented just the querier and sidecar component of thanos, to enable HA(fill gaps in grafana dashboards) for a 2 node prometheus. Since we are executing a very heavy query the load was considerably high on one of the prometheus node, before implementing thanos, now querier fanouts the query to both the nodes and it is creating similar high load on both the nodes compared to what was on just one. prometheus, version 2.13.0 |
Thanos, Prometheus and Golang version used
What happened
The query provided below timeouts (after the default 2 minutes query timeout), but before it does that, Prometheus gets OOM.
What you expected to happen
Either the query to complete successfully, or timeout (assuming it's too complex) without bringing Prometheus down.
How to reproduce it (as minimally and precisely as possible):
We are using the following command to perform the query:
curl "https://thanosquery/api/v1/query_range?query=label_replace(%0A%20%20histogram_quantile(%0A%20%20%20%200.999%2C%0A%20%20%20%20sum(%20%20%20%20%20%20rate(join_joined_left_vs_right_timestamp_diff_bucket%7Bpartition%3D~%220%7C1%7C2%7C3%7C4%7C5%7C6%7C7%7C8%7C18%7C19%7C20%7C21%7C22%7C23%7C24%7C25%7C26%7C27%7C28%7C29%7C30%7C31%7C32%7C33%7C34%7C35%22%2Cmarathon_app%3D~%22%2Fsome%2Fapp%2Fregion%2Fjoin.*%22%7D%5B5m%5D)%0A%20%20%20%20)%20by%20(le%2C%20marathon_app%2C%20partition)%0A%20%20)%2C%20%0A%20%20%22region%22%2C%20%22%241%22%2C%20%22marathon_app%22%2C%20%22%2Fsome%2Fapp%2F(.*)%2Fjoin.*%22%0A)&start=1532343000&end=1532948400&step=600"Running this once or twice, always brings our nodes down. If we run the same query directly against Prometheus (not via Thanos Query), it completes quite fast successfully (we store 24h data on Prometheus).
Anything else we need to know
We are running independent Docker hosts with the following containers:
Thanos Compactor is running independently on a different host. We also try running Thanos Store on a different host than other containers.
One thing to note, that if we have multiple (let's say have 4 nodes) this configuration and put Thanos Query under an Nginx-based load balanced, a query to one of the Thanos Query instances brings all 4 hosts down (due to all 4 Promehetues getting OOM).
Environment:
Storage:
We are not sure whether the issue is caused by Prometheus or Thanos. Maybe it's somehow directly related to our setup (for example
join_joined_left_vs_right_timestamp_diff_bucketmetric having a huge number of different labels, which results into a large number of different timeseries that cannot be handled when running the mentioned query). Anyhow, any guidance or tips would be really appreciated.The text was updated successfully, but these errors were encountered: