Add filtering on Thanos Receiver

[piclemx]

Is your proposal related to a problem?

We were looking at adding Thanos with the receiver patterns (router + ingestor). Everything seems to be good for us, except that we need to have more control on the metrics that will be ingested inside Thanos. One of those reasons is that we want to make sure metrics doesn't have high cardinality and make it hard at the end for thanos to scale (p.s might have other ways). The second one is that we want to ingest metrics of multiple applications, but we are not interested in all those metrics.

In that case, I think the Thanos Router Receiver could archive that quite easily. Even before it reach the ingestor, the router could drop metrics that are not useful and can add more protection on the receiver ingestor.

Describe the solution you'd like

Maybe adding a flag or a config to the thanos receiver to filter out metrics that we don't want to keep in the TSDB or being flush to the Object storage

Prometheus (multiples) --> Thanos Router receiver (filter out unwanted metrics) --> Thanos Receiver Ingestor.

We could use the RelabelConfig to drop metrics inside Prometheus, but that means we give the control to the integrator and not us.

Describe alternatives you've considered

Some alternatives that we were looking for.

Use the remote write protocol with the Prometheus Kafka Adapter. Sending metrics on a kafka topic. Having something like Benthos (Pipeline processor) to read from it and drop message that are not whitelisted and send them after to the Thanos Receiver Router.

[piclemx]

Thanks, @yeya24, for adding this. I was looking to do that this week, but seems you have already taken care of this 😄.

Do you know when this could be release, so we can test it more?

[yeya24]

@piclemx I don't have a detailed date for the next release. But you can always test against that feature branch or main branch once it is merged.