Add and support receiver creator endpoint properties

[rmfitzpatrick]

Description:
Adding a feature - These changes add a new boolean accept_endpoint_properties config field and mechanism to evaluate remotely specified receiver template configuration by observer endpoint environment content (disabled by default). This feature will allow the receiver creator to add or modify evaluated receiver template config, rules, and resource attributes specified by the applicable observer metadata for the containing endpoint alone.

Link to tracking Issue:
#17418

Testing:
Added and updated tests suites.

Documentation:
Updated readme to describe feature.

[dmitryax]

Couple Qs for clarification:

• What properties do I need to set to scrape metrics only from the annotated (let's say redis) pods and ignore other redis pods that are not annotated?
• Does removing annotations stop metrics scraping?

[rmfitzpatrick]

• What properties do I need to set to scrape metrics only from the annotated (let's say redis) pods and ignore other redis pods that are not annotated?

The rule/config/resource_attributes* will only be applied to the endpoints* that contains the properties, so it doesn't transfer to any other endpoint evaluation from an unannotated redis pod. This occurs in observerHandler.OnAdd() for each observed endpoint.

• Does removing annotations stop metrics scraping?

Yes, since the observerHandler.OnChange() method removes the target receiver with OnRemove() and then runs OnAdd() with the updated endpoint no longer containing the annotation, its properties aren't able to be evaluated again so only the service config would apply.

[dmitryax]

The rule/config/resource_attributes* will only be applied to the endpoint that contains the properties, so it doesn't transfer to any other endpoint evaluation from an unannotated redis pod. This occurs in observerHandler.OnAdd() for each observed endpoint.

If it's only applied to the endpoint containing the properties, why do we need to transfer the recievercreator's endpoint discovery configuration to the endpoint itself? I'm talking about the rule annotation. Shouldn't it be just a boolean annotation or just the presence of any other receiver-creator annotation?

[rmfitzpatrick]

If it's only applied to the endpoint containing the properties, why do we need to transfer the recievercreator's endpoint discovery configuration to the endpoint itself? I'm talking about the rule annotation. Shouldn't it be just a boolean annotation or just the presence of any other receiver-creator annotation?

For port endpoints, the annotations come from the sub-pod, so a default true would cause a hit for all ports (all container ports), when likely a single is desired. Similarly for container endpoints, an endpoint exists for each port (endpointForPort). This way specifying the port in the* type == "port" and port == xyz or type == "container" and port == xyz rules* is helpful.

edit: In cases where a receiver's default configuration leads to telemetry, just setting the rule label/annotation is all that's needed.*

[dmitryax]

For port endpoints, the annotations come from the sub-pod, so a default true would cause a hit for all ports (all container ports), when likely a single is desired. Similarly for container endpoints, an endpoint exists for each port (endpointForPort). This way specifying the port in the* type == "port" and port == xyz or type == "container" and port == xyz rules* is helpful.

Shouldn't port be provided as part of config? Configuring an endpoint discovery rule on the endpoint itself seems a bit confusing.

[rmfitzpatrick]

Shouldn't port be provided as part of config? Configuring an endpoint discovery rule on the endpoint itself seems a bit confusing.

I think this may be a general receiver creator use question: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/receivercreator#rule-expressions. The port variable in an environment can be used in an arbitrary place in a value* in config, but is also part of the endpoint identity. Because there is an endpoint generated for each port, all of which share labels/annotations from an env, having the port as an identifier (being included in a rule)* acts as a filter.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[hughesjj]

LGTM

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[atoulme]

Probably can peel off this logger change in a separate PR just to make the diff smaller

[atoulme]

move it up to around line 70 to keep it close to the struct definition?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.