Cannot inject envs to targetAllocator

[sumo-drosiek]

We have several serviceMonitors, which are using environment variables, e.g:

spec:
  endpoints:
  - port: otelcol-metrics
  namespaceSelector:
    matchNames:
    - $(NAMESPACE)
  selector:
    matchLabels:
      sumologic.com/app: fluentd-metrics
      sumologic.com/scrape: "true"

It would be nice to add and support env in targetAllocator so we could seamlessly migrate from Prometheus to OpenTelemetry

[swiatekm]

This is a bit tricky, as this substitution is done by config reloader in prometheus-operator managed configurations. See: https://github.com/thanos-io/thanos/blob/main/pkg/reloader/reloader.go#L82.

This means that the variables are substituted in Prometheus' environment, which maps to the otelcol environment in our case. With TargetAllocator enabled though, Prometheus receiver gets this configuration directly from the TA. Doing the substitution in Prometheus receiver itself would represent a break from Prometheus upstream semantics.

We could do the substitution in TA, using TA's environment. This seems fairly straightforward to do if we want to, the more difficult question is whether we should. On one hand, we'd be more compatible with prometheus-operator, which is a pretty big deal considering this is the primary purpose of TA. On the other hand, it wouldn't be exactly the same thing, and the ambiguity could trip some people up. Technically, we could automatically mirror the env variables from the collector to the TA, and be fully compatible that way, but that would make this feature a lot more complicated.

I'm cautiously in favor of doing it in the TA in a naive way and documenting it.

[gracewehner]

+1, my team would also like this and are ok with it being in the TA

[jaronoff97]

@KitSutliff is going to take this one! 🥳

[swiatekm]

#1875 is not enough to resolve this issue. It did add the ability to define env variables in the Target Allocator, but not variable expansion for the configuration.

[KitSutliff]

great catch, I can add a couple lines to address it next week