Help with thanos on huge prometheus environment #569
Comments
|
Thank you for this write up! So many details <3 Some initial questions:
Are you describing current setup? If yes, Which of those prometheuses are actually uploading to S3? All? And they have 30 days locally retention as well? Is local compaction enabled?
Awesome, super useful data. Sounds like a valid bug to me, let's think how to improve this.
Do you mean for query purposes? Do you have numbers how much it takes during idle time?
Interesting observation, wonder if that hits downsampled data. The panic itself has to be fixed. Did not have time to dig into it properly.
Nice finding I feel it is because you have lots of series, we cannot downsample that dimensions. Those numbers are for biggest 2w blocks? Re 4. Yea trickster is not ideal - but have you looked on query distribution, where the latency comes from? Have you looked on max concurrent queries flag that is available for thanos-query as well as Prometheus? If we are missing some instrumentation there, we need to add this as well. We actually hit recently same issue, upping the max concurrent queries, helped a lot. Also we fixed this: #563 Thanks for writing it up in details, we really appreciate the feedback. We also appreciate contributions if you want to improve this quicker (((: Cheers! |
|
Thank you for quick response )
Yes the all upload to s3 because the have different type of metrics, and only in s3 they all in one place
i think its possible to use tmp prefix for new not fully created data block in s3 and then rename it, but not sure if its possible with s3, and this does not fix situation when compacted old blocks is deleted by compactor and do not synced properly by s3, but maybe its different problem not sure.
Not only query, mostly it consume max memory during initialization of s3 bucket where we have 900GB of data blocks. During idle it mostly use 160-200GB with this backet.
This was using data that are not downsampled by thanos, it was uploaded by hand from prometheus just to store it in s3. But we sometimes we have this problem with small data that already compacted and downsampled by thanos, not sure why its acquired yet.
No this blocks have data only for 2 days. This blocks was compacted from 2h blocks by compactor and then on second day they start downsampled, but the size still almost the same, i just wonder if this is normal, because i believed that it will be much smaller than original block
Yes i tried to use this option and increase it to 300 concurrent queries, but looks like this problem is because huge number of data sources and connection speed, i will try to run query and store with some of prometheus servers on one server to test if this will improve the query speed |
So are you saying that your Grafana -> thanos-query asks
Worth to grab some traces maybe? |
|
Sorry i was not clear. Before thanos we use local compaction by prometheus itself with max.tsdb=1d and min.tsdb=2h. Before running thanos we tested if we can upload all existing data blocks compacted by prometheus. During init of store component thanos consume near 200-250GB of RAM to load 900GB of metrics. This setup used only one static configured store on query component and this store was s3. After this we test if we can use this metrics by running only thanos. And it worked well, but sometimes it falls when run queries bigger than 30 days interval, but maybe it falls because of other things not sure. |
|
Test prometheus and store component in AWS with s3 store and it working well almost all time, so looks like its slow network issue, will close this issue for now. Also have last question, @bwplotka does it make sense to make local storage(blocked device) for thanos that will be synced from s3 if needed some speedup |
All of those issue are fine, or what exactly? (: Just want to make sure all is covered.
Can you be more specific? To what thanos component? sidecar? |
|
for 1. issue I created ticket: #564 |
I mean all components, i deploy prometheus with thanos store and query in aws and it worked fast and stable
|
Nice! For context, where you have been running before, when you were having this issues?
That makes sense only for small number of blocks. The main advantage of using object storage is that it is cheap and reliable. Downloading ALL blocks in store gateway (when only very smal % will be actually queried) is waste of your disk and time. Also it is not cheap to store petabytes of data on your local storage (: and you can store that many data on object storage just fine, with relatively quick access. |
|
Ok, I think it's time to close this issue as it's not really actionable, plus very old. We improved many things from last year and we are rewriting store GW block loading #1471 so things are changing here (: Let's add issue / notes to existing issue for each item separately if still problematic. Let me know if that makes sense! |
|
FYI, the following article could help understanding better Thanos issues on large-scale Prometheus setup. |
|
I don't think there is anything wrong with running Thanos on huge Prometheus environment. If only, Thanos makes it much easier. The main reason for this issue looked like "high memory consumption". Issue was claimed a long time ago for v0.1.0 version of Thanos (!). Plus we are working on improving it even more here: #1471 (: I wouldn't also personally suggest running remote-write streaming for a high volume of data, but definitely it's worth to know your options. Thanos now supports remote write as well: https://thanos.io/proposals/201812_thanos-remote-receive.md/ |
We need you help, and it will be much appreciated if you could look at this problems, give some advice how better build environment using Thanos and our infrastructure or maybe how to properly configure it. We can give any type of logs or info you need.
There are several problems we would like to address with Thanos:
We use Thanos with 5 prometheus servers with following structure:
We have several problems after almost one week of testing with production metrics:
We tried to play with configuration of parameters
indexCacheSizeandchunkPoolSizebut no luck its still failed sometimes. As I understand its failed mostly when try to run query for metrics over long time range, more than 30 daysoriginal block
01CSF730JSJR3Z9EJXXFZTJTR1and compacted
01CSGX47CZCX85NK622SSWQ1CZIn order to speedup process we tried to use trickster as cache between grafana and Thanos, and it helps a lot, it is still 2+ times slower, but there is another issue if one of store is down it will cache wrong result and will not overwrite it if store is up again. This issue is more to proxy authors. This information is just to give you an idea why solving problems blocks us, and simple caching on trickster did not work out.
Having all this problems we are blocked with moving on to production, as we anticipate much more problems when many people will start to query and run grafana dashboards simultaneously.
The text was updated successfully, but these errors were encountered: