querier: Federated Thanos Targets Page

[d-ulyanov]

Hey, Thanos team :)

It would be great to have Prometheus-like "targets" page but from all connected Prometheuses.
From time to time it's pretty inconvenient to spend time finding which Prom scrapes some target, especially if your Prom setup is big (we have ~30 instances).
It would be much easier to filter targets on a special page instead of writing query up{service=~...}.

What do you think, guys?

[GiedriusS]

My 2 cents: I would love this feature but only if Prometheus had an API which exposed all of this information. AFAICT it doesn't exist ATM. I wouldn't want to attach ourselves to the up metric because:

• Its name might change
• It does not expose labels which were on the target before relabelling such as __scheme__.

WDYT?

[2nick]

The documentation describes api endpoint to get information about targets https://prometheus.io/docs/prometheus/latest/querying/api/#targets which expose values for both types of labels - discovered and formed by job relabeling.

[GiedriusS]

@2nick thanks for looking into it (I haven't when I wrote the original comment)! Then we could probably make a new method in the StoreAPI which returns these (if any exist) and Thanos Query could have this information, and show it in the web UI.

[bwplotka]

Nice ideas!

So the use case you provided is:

it's pretty inconvenient to spend time finding which Prom scrapes some target, especially if your Prom setup is big (we have ~ 30 instances).
It would be much easier to filter targets on a special page instead of writing query up{service=~...}.

Why query is not acceptable? What about having Grafana dashboard for this? (:

Then we could probably make a new method in the StoreAPI which returns these (if any exist) and Thanos Query could have this information, and show it in the web UI.

Disagree - StoreAPI is a generic form of getting metric data in an efficient way. Targets strictly relate to pulling based collectors/scrapers. This means that for now only Prometheus would be implementing this (or Thanos sidecar). I think if anything, it will have to be another gRPC service.

However, I mentioned some solution - Grafana dashboard (: As you can track every metric to the Prometheus/Metric source thanks to external labels. Would that solve your use case @d-ulyanov ?

[2nick]

@bwplotka

Disagree - StoreAPI is a generic form of getting metric data in an efficient way. Targets strictly relate to pulling based collectors/scrapers. This means that for now only Prometheus would be implementing this (or Thanos sidecar). I think if anything, it will have to be another gRPC service.

So you'll be OK if such functionality will be implemented as part of Thanos sidecar but as particular gRPC service with it's own API (maybe smth like "PrometheusProxyAPI")?

Because it does not look like that is possible to get "labels before relabeling" using only metrics data and this info is really valuable for debugging some types of issues/investigations.

[bwplotka]

Because it does not look like that is possible to get "labels before relabeling"

maybe, but do you need this? You only need to find out the correct Prometheus that was the source and compose a link to target page of it. All of this can be done on Grafana dashboard, including HTML link.

[brancz]

I have mixed feelings about this. I feel a combination of Prometheus providing more/better logs about scrapes in combination with the up metric might be a better fit than a "federation" approach. At the end of the day if you need information from the edge, then the edge is likely to be your best source :)

[2nick]

Yes, we are actively using additionalScrapeConfigs to add jobs for non-k8s targets with custom relabelings/certificates/other options which can't be configured with ServiceMonitor and it's really useful information for debugging some cases.

Also we can implement better targets page to have a more convenient way to work with targets then browser search-by-page (like rich UI with nice seach/filter-by-labels/labels view) and we think that Thanos query can be a good choice to make a single place to ease understanding of infrastructure and investigating issues for really complex prometheus installations.

For example at the moment we have like 4 environments with role-based (5 roles), sharded (each role has 4 shards) and replicated (each shard has it's own replica) Prometheus instances to collect not only k8s applications but also some software which is installed on bare metal servers (win/*nix). :)

[d-ulyanov]

@bwplotka @brancz @GiedriusS thanks for your replies, guys.
Targets page could be cool for ordinary users (who just using querying metrics, not setting up Thanos :)) because switching from Prometheus to Thanos would seem more seamless for them.
In spite of this, I agree that StoreAPI won't be a good place for such logic and we could add more

Grafana dashboard is a disputable solution here because in this case we should explain to our users where they should go to check their targets :)

Additional GRPC service sounds reasonable for us, WDYT @bwplotka ?

[bwplotka]

Thanks. Especially the input for the use cases and user experience is quite useful!

Grafana dashboard is a disputable solution here because in this case we should explain to our users where they should go to check their targets :)

Before Thanos, you needed to do that as well right? E.g `hey, it seems like this metric is from cluster=XYZ. So please go to "http://prometheus.xyz.example.com:9090/targets" to see what's wrong with your scrape configuration?

We are not saying "no" but we are trying to understand first what's the best user and operator experience here. For example:

• Targets and new Service Discovery pages are evolving in Prometheus. Our API would need to be maintained for that.
• Thanos can be extended to support any other "scraper" or "collector" with StoreAPI and TSDB format of blocks. What if someone adds some other source of metrics? One of the sources is also https://thanos.io/components/rule.md/ - it is producing some metrics for recording rules and ALERTS{...}. We need to be careful here as such produced metric is affected by dependent metric (e.g from some Prometheus), so it's quite complex here (: In fact, the same for potential Grafana dashboard.

Need to think about it more.

What if Store page would have links user can customize that will forward to Prometheus UIs?

[d-ulyanov]

Okay, I agree with @bwplotka, Targets page adds pretty controversial Prometheus functionality to Thanos. Let's close this issue at the moment, we'll use the dashboard or smth like that.
Thanks all for your comments!

[bwplotka]

Targets page adds pretty controversial Prometheus functionality to Thanos.

cc @d-ulyanov

While it being controversial, we actually started work on Federated Rules API #2200 (Proposal to come! cc @s-urbaniak), so might want to like on Federated Targets as well in similar way ❤️

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Still needed (: and planned to do! Help wanted

…

On Sun, 17 May 2020 at 22:05, stale[bot] ***@***.***> wrote: Hello 👋 Looks like there was no activity on this issue for last 30 days. *Do you mind updating us on the status?* Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command <https://probot.github.io/apps/reminders/> if you wish to be reminded at some point in future. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1375 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O57NNPAHB2I2TGOJO3RSBGS3ANCNFSM4IJWTNEQ> .

[brancz]

fwiw I've also come around to agree that with deduping functionality this actually fits pretty well in Thanos

[yeya24]

#3350

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

PR is almost there, #3350

[GiedriusS]

We have the API, now it's just a matter of adding a new page to Thanos Query. I propose something like "Federated Targets" or something (:

[2nick]

@GiedriusS you won't believe, but in "New UI" targets page is fully functional :)

I tested my api with it, simply typing http://localhost:9090/new/targets

[onprem]

Haha cool. I am not that surprised tbh because it will work as long as we follow the Prometheus API (same route, same data format). Looks like the only work remaining is to add the link to Querier's nav bar. :)

[bwplotka]

Wooo! Amazing! 🤗

So what is left to do?

only work remaining is to add the link to Querier's nav bar. :)

Isn't that simpler to make React UI a new default? Which will solve this too?

[onprem]

Isn't that simpler to make React UI a new default? Which will solve this too?

By adding the link in querier's navbar I meant navbar in the new UI. We have the page working it's just that we are not advertising it. Created #4045 to fix this :)

[yeya24]

I believe this one is done. Close now.