SDK YAML (Declarative) Configuration is Not Working

[shaiajneh]

I have been trying to use the SDK Configuration file but the agent refuses to start and its unable to read the resource attributes. I am using the same example configuration file available here and the following repo for demo purposes.

The only env variable I am using is OTEL_EXPERIMENTAL_CONFIG_FILE which points to the config file.

order-service-1  | OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
order-service-1  | [otel.javaagent 2024-10-03 15:47:39:233 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.8.0
order-service-1  | OpenTelemetry Javaagent failed to start
order-service-1  | io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: Unable to parse configuration input stream
order-service-1  |      at io.opentelemetry.sdk.extension.incubator.fileconfig.FileConfiguration.parse(FileConfiguration.java:107)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
order-service-1  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
order-service-1  |      at java.base/java.lang.reflect.Method.invoke(Method.java:569)
order-service-1  |      at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.maybeConfigureFromFile(AutoConfiguredOpenTelemetrySdkBuilder.java:551)
order-service-1  |      at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:431)
order-service-1  |      at io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller.installOpenTelemetrySdk(OpenTelemetryInstaller.java:29)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:122)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:102)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
order-service-1  |      at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
order-service-1  |      at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
order-service-1  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
order-service-1  |      at java.base/java.lang.reflect.Method.invoke(Method.java:569)
order-service-1  |      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
order-service-1  |      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
order-service-1  | Caused by: java.lang.IllegalArgumentException: Cannot deserialize value of type `io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.Attributes` from Array value (token `JsonToken.START_ARRAY`)
order-service-1  |  at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.OpenTelemetryConfiguration["resource"]->io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.Resource["attributes"])
order-service-1  |      at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4624)
order-service-1  |      at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:4555)
order-service-1  |      at io.opentelemetry.sdk.extension.incubator.fileconfig.FileConfiguration.parse(FileConfiguration.java:115)
order-service-1  |      at io.opentelemetry.sdk.extension.incubator.fileconfig.FileConfiguration.parse(FileConfiguration.java:105)
order-service-1  |      ... 22 more
order-service-1  | Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.Attributes` from Array value (token `JsonToken.START_ARRAY`)
order-service-1  |  at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.OpenTelemetryConfiguration["resource"]->io.opentelemetry.sdk.extension.incubator.fileconfig.internal.model.Resource["attributes"])
order-service-1  |      at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
order-service-1  |      at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1767)
order-service-1  |      at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1541)
order-service-1  |      at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1488)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeFromArray(BeanDeserializer.java:660)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:211)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
order-service-1  |      at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
order-service-1  |      at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
order-service-1  |      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
order-service-1  |      at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:4619)
order-service-1  |      ... 25 more

[jaydeluca]

i think the issue here is because the example file you reference is in the format for the recently released configuration v0.3.0, which had some breaking changes. This new version of configuration was only just recently merged into the sdk a few days ago, and is not yet available in the java agent. I imagine that will land soon.

In the meantime, if you want to continue with your example, I believe you can make a few modifications to your configuration file and it should work.

Change the attributes block from:

attributes:
  - name: service.name
    value: unknown_service

to

attributes:
  service.name: unknown_service

And then for each occurrence of a headers array input, change from:

headers: []

to

headers:

or

headers:
  api-key: 1adf23d4

[shaiajneh]

Thanks a lot @jaydeluca
I have tried your recommendaiton and used the config file in the example repo. Which follows the same convention you mentioned.
Here is the config I am using.

OTEL_EXPERIMENTAL_CONFIG_FILE: "/otelagent/sdk-config.yaml"

file_format: "0.1"

resource:
  attributes:
    service.name: agent-example-app

logger_provider:
  processors:
    - batch:
        exporter:
          otlp:
            endpoint: http://123.123.213.122:4317
            protocol: http/protobuf

tracer_provider:
  processors:
    - batch:
        exporter:
          otlp:
            endpoint: http://123.123.213.122:4317
            protocol: http/protobuf

meter_provider:
  readers:
    - periodic:
        exporter:
          otlp:
            endpoint: http://123.123.213.122:4317
            protocol: http/protobuf
  # Example of how to use view to drop a metric which isn't needed.
  # This configuration is not available with the environment variable configuration scheme.
  # NOTE: most users will want jvm.memory.limit metric and should remove this view.
  views:
    - selector:
        instrument_name: jvm.memory.limit
      stream:
        aggregation:
          drop:

While using the latest agent 2.8.0, it fails to start because config is null

order-service-1  | [otel.javaagent 2024-10-06 11:06:00:555 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.8.0
rder-service-1  | OpenTelemetry Javaagent failed to start
order-service-1  | java.lang.NullPointerException: Cannot invoke "io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties.getString(String)" because "config" is null
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentInstaller.copyNecessaryConfigToSystemProperties(AgentInstaller.java:204)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:126)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:102)
order-service-1  |      at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
order-service-1  |      at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
order-service-1  |      at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
order-service-1  |      at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
order-service-1  |      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
order-service-1  |      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
order-service-1  |      at java.base/java.lang.reflect.Method.invoke(Method.java:569)
order-service-1  |      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
order-service-1  |      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)

Now if I use an older version of the agent 1.33.3 It starts connecting to the specified endpoint

order-service-1  | [otel.javaagent 2024-10-06 11:08:05:762 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.33.3
order-service-1  | [otel.javaagent 2024-10-06 11:08:18:103 +0000] [OkHttp http://123.123.213.122:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: timeout
order-service-1  | java.io.InterruptedIOException: timeout

Another thing I have noticed is when switching the env variable (using version 2.8.0)

# FROM
OTEL_EXPERIMENTAL_CONFIG_FILE: "/otelagent/sdk-config.yaml"
# TO
OTEL_CONFIG_FILE: "/otelagent/sdk-config.yaml"

The error changes but the agent still tries to connect to default values

order-service-1  | [otel.javaagent 2024-10-06 11:11:14:506 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.8.0
order-service-1  | [otel.javaagent 2024-10-06 11:11:14:532 +0000] [main] WARN io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder - otel.config.file was set, but has been replaced with otel.experimental.config.file
order-service-1  | [otel.javaagent 2024-10-06 11:12:04:665 +0000] [OkHttp http://localhost:4318/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4318

Is the latest agent not compatible with any of formats?

[jaydeluca]

I apologize, I gave you incorrect information. There is one more PR that needs to be merged before the agent will have support for declarative config

[shaiajneh]

Perfect! Thanks a lot @jaydeluca

[jaydeluca]

@shaiajneh now that 2.9.0 is out, this should work. See this PR for reference

[shaiajneh]

@jaydeluca
It works thanks!