Service Discovery: Adding generic HTTP service discovery

[matej-g]

Is your proposal related to a problem?

Currently, there is no generic service discovery mechanism in Thanos, that would function remotely. There is currently file service discovery, which however has some legitimate limitations - if a user wishes to dynamically change the content of the file / services, this requires a sidecar process which updates the configuration file. Such pattern might, however, not be applicable in all environments. Having a mechanism to pull changes from a remote place instead would alleviate the issue in such cases.

Related issues:

• Add other service discovery support to Thanos #3538
• Add support for "consul_sd_configs" for thanos rule. #4313

Describe the solution you'd like

The proposal is to introduce a generic HTTP service discovery, which could be based on regular polling of an endpoint, which would contain information on available services, in a pre-defined format.

Describe alternatives you've considered

• One mentioned alternative is to generate SD files dynamically via a sidecar, but this is somewhat cumbersome and might not be possible in all scenarios (requires employing an extra process with access to the underlying file system)
• Another would be to enhance Thanos each time with particular SD service discovery mechanisms that Thanos users propose. This could however lead to 'opening the flood gates', where the number of SD service discoveries could grow uncontrollably (which has already been discouraged by the maintainers), leading to dependency hell and bloating of the amount of code. Offering a generic remote-based (HTTP) service discovery is a compromise between having to operate on the local file system and allowing for large number of provider-specific service discoveries into the code base.

Additional context

• This suggestions comes on the back of another proposal to create a community-powered service discovery sidecar, see https://docs.google.com/document/d/1PinFPQLwa4uV0Ylj2maRRNRx2tA6TOMcqPQxUzcHdkA/edit#heading=h.26vnob2480d2. Besides having HTTP service discovery on its own merits, it would incorporate well into such community-friendly solution, where users could choose between support for 2 generic solutions backed by the sidecar (either file or HTTP SD)
• Generic HTTP service discovery has also been recently introduce into Prometheus with similar reasoning (proposal: https://docs.google.com/document/d/1tVeuzjpU4-TiYPNWJXKmcyIuZF6A2tUq270RbBT5zho/edit#, PR: Generic HTTP sd prometheus/prometheus#8839). Introducing HTTP SD would mean expanding on the processes that currently exist in our ecosystem and that our users are familiar with.

cc @saswatamcode

[saswatamcode]

Thanks a ton for writing this up! It's a great idea and would fit really well with the SD sidecar. 🙂

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[matej-g]

Still needed ⬆️

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[tohjustin]

@matej-g @saswatamcode I'm interested in implementing HTTP SD on Thanos. Should I proceed with writing a proposal document or that isn't necessary?

[matej-g]

Hey @tohjustin, thanks for your interest! It would be awesome if you'd like to tackle this.

I believe this issue can serve as a proposal (or at least a base for it). As long as we have clear understanding and agreement, you could go directly into writing an implementation (and we can turn this issue into a formal proposal, if necessary, but that does not block work on the actual feature).

Thoughts @thanos-io/thanos-maintainers?