TargetAllocator for daemonset

[tcolgate]

Ideally, when deployed as a daemonset, collectors would scrape the pods running on the node
local to the daemonset instance. It's not obvious from the docs or configs as to if that is achievable
with the current targetAllocator strategies.

Am I missing something? Would you accept a PR that implements this?

[m1rp]

We're also interested in this.
we've been reading through the docs and tried a few things but couldn't get this to work.

[swiatekm]

I'm curious if your use cases includes Prometheus Custom Resources as well. Do you want each Otel Pod to scrape Pods on the same Node, as defined by a ServiceMonitor or PodMonitor? Or just all Pods on the Node with the right annotation? I believe you can achieve the latter with plain Prometheus configuration by using selectors for a pod role.

[tcolgate]

Yeah, we are using ServiceMonitor as it seems to be fairly common for third party charts. Most things I've seen still end up with the older style scrape labels, so I could maybe just use those. I'll have a look around.

[jaronoff97]

yeah this is a good idea, we would ultimately implement this as a new strategy that can only be used when target allocator's mode is daemonset. We would first implement the strategy and then expose it under the new target allocator CRD as part of our v2 milestone.

[swiatekm]

@jaronoff97 do we need to tie this to v2? It's not a breaking change and doesn't have any dependencies.

[jaronoff97]

Its something I would love to get done in the TA prior to the v2 so that when we make the target allocator CRD it can start with a mode which right now it doesn't have.

[jaronoff97]

@matej-g is going to take this on! Thank you 🙇

[matej-g]

One more gap I have identified in #2430 (comment) - if we encounter a target we cannot assign to any node, how should we handle it. As @swiatekm-sumo suggested:

• Ignore these targets
• Assign them randomly
• Assign them according to some deterministic scheme (could be what the consistent hashing strategy uses)

What my current implementation doing and what I was leaning towards was to ignore these targets, as I would expect users to use this strategy only scenario where we can detect nodes for targets (i.e. they use this strategy only in conjunction with Prometheus CRs or Kubernetes SD).

[swiatekm]

I think ignoring the targets is fine as long as we log a warning about it. Possible edge cases I can see for it are Kubernetes SD roles which don't actually add the node label, like service. These can be used to monitor control plane components exposed as Services, but as long as we document this limitation and log warnings about it, it should be fine.

[jaronoff97]

I agree with @swiatekm-sumo, ignoring targets without this fine. However, we should probably log per job not per target. Logging per target could get very spammy.

The idea of assigning them by using a fallback strategy is an interesting one, but something we should look in to as a follow up IMO.