Unclear merge order for Resources from environment variable

[Aneurysm9]

It is unclear from a reading of the specification regarding resource information from environment variables the order that those detected resource attributes should be merged with resource attributes provided by the application. The spec states:

The SDK MUST extract information from the OTEL_RESOURCE_ATTRIBUTES environment variable and merge this, as the secondary resource, with any resource information provided by the user, i.e. the user provided resource information has higher priority.

The trouble I'm having is determining what "user provided resource information has higher priority". I would expect that the resource attributes detected from the environment, as late-binding data, would take precedence as it gives the operator-user the opportunity to override defaults provided by the devloper-user in the source code. Even in the case of run-time resource detection, allowing the environment-specified resource attributes to take precedence allows operator-users to override faulty or unwanted resource attributes coming from those detectors that they may not be able to alter via configuration.

That said, others in the Go SIG have interpreted this passage to require that resource attributes from the environment must be subordinated to any resource attributes provided by the developer-user through explicit resource construction or the use of detectors. I can see how the language currently in the specification can be interpreted in either way and suspect that this latter interpretation is what is intended.

Clarification of the language would be appreciated and, if this latter interpretation is intended, perhaps allowing for an operator-user to override this ordering through an additional environment flag could be considered. It seems like a significant reduction in flexibility compared to other environment-based configurations, such as exporter targets, that are intended to override defaults provided in code.

[Oberon00]

That said, others in the Go SIG have interpreted this passage to require that resource attributes from the environment must be subordinated to any resource attributes provided by the developer-user through explicit resource construction or the use of detectors. I can see how the language currently in the specification can be interpreted in either way

I think the language is pretty clear on that part. If you have a conflict between A and B and B has higher priority then B wins. So the user-provided resources override the environment variable. However, the question is then: (1) Does the environment variable not count as user-provided? (2) What does user-provided mean? Hardcoded?

[Aneurysm9]

However, the question is then: (1) Does the environment variable not count as user-provided? (2) What does user-provided mean? Hardcoded?

Yeah, I think this is the crux of my issue. It seems odd to not consider environment variables as user-provided and very odd to not allow their use for overriding hardcoded values.

[carlosalberto]

Assigning this to @Aneurysm9 in order to start gathering scenarios for this (and potentially have @jsuereth help with the write-up portion ;) )

[jsuereth]

I'm in. For context, we had questions in general around resource detection ordering too: GoogleCloudPlatform/opentelemetry-operations-java#92

[Aneurysm9]

These use cases assume the application developer has constructed a non-default resource and that SDK implementations merge application-developer provided resources with resources detected from environment variables at the time that a TracerProvider or MeterProvider is initialized. The question is in what order should the resources be merged? Assuming arguments to resource.Merge() are in ascending priority order - that is, values from later arguments will override values from earlier arguments with the same key - which of the following is correct?

finalResource := resource.Merge(resource.Environment(), appResource)

Or

finalResource := resource.Merge(appResource, resource.Environment())

In the following use cases I would expect the latter to be used. The resource attributes from the environment should be late-binding and take precedence over any resource attributes specified in the application at build time or even attributes detected by a detector configured at application build time.

• Specifying service name at runtime
  • Application constructs a resource with a static service name
  • Operator wishes to deploy multiple different configurations of the application and have each report a distinct service name
• Rectifying an incorrectly detected resource
  • Application constructs a resource with one or more detectors included
  • A detector sets a resource attribute to a value an operator feels is incorrect
  • Operator wishes to use the environment to assign a different value to that attribute

These two use cases seem to cover a broad range of situations where an operator might have better information about the entity producing telemetry than application or detector developers who may be far removed from the entity’s runtime environment.