OTel Spring starter - Automatically add the OTel log appenders

[jeanbisutti]

See #8920 (comment)

[jeanbisutti]

Two topics:

• Add OTel log appender only if no otel appender is found (for example from XML config)
• Programmatic configuration of the OTel log appenders

[jeanbisutti]

I took a look for Logack. The implementation could be based on Configurator. See also DefaultJoranConfigurator.

I could help to verify it works with GraalVM native.

[jeanbisutti]

I took a look for Logack. The implementation could be based on Configurator. See also DefaultJoranConfigurator.

I could help to verify it works with GraalVM native.

In this way the Logback appender would be configured before the Spring initialisations and the Spring logs should be appended.

[jeanbisutti]

I took a look for Logack. The implementation could be based on Configurator. See also DefaultJoranConfigurator.

I could help to verify it works with GraalVM native.

This implementation may not work well with the Spring Boot initialisations (see AbstractLoggingSystem). An alternative implementation may be to add the OTel Logback appender (if it is not declared in an xml file) from a new logging system (see LogbackLoggingSystem and this configuration).

[zeitlinger]

Here's an approach I took with another spring starter: https://github.com/grafana/grafana-opentelemetry-starter/blob/d0a762e8d4fc27436db04b0e0a5ff77d1209d01a/src/main/java/com/grafana/opentelemetry/LogAppenderConfigurer.java#L5

[jeanbisutti]

Here's an approach I took with another spring starter: https://github.com/grafana/grafana-opentelemetry-starter/blob/d0a762e8d4fc27436db04b0e0a5ff77d1209d01a/src/main/java/com/grafana/opentelemetry/LogAppenderConfigurer.java#L5

@zeitlinger Thanks for sharing. With this approach, the OpenTelemetry appender is added during the creation of the OpenTelemetry bean. However, some logs can be emited before this. The two other approaches I mentioned above should not have this limitation (if they work).

[zeitlinger]

I remember that we discussed the challenge of having an otel instance early enough - because beans are not available at the early stage when logs can be emitted.

IIRC, you'd have to cache the log lines from this sage.
From the links, I couldn't see how you propose to address this challenge.

[jeanbisutti]

Today, users must declare the OpenTelemetry appender in an xml file (logback.xml, logback-spring.xml or log4j2.xml). So, the OpenTelemetry appender is created and configured early at application startup.

The logs placed in a queue (see this for Logback and this for Log4j2) until an OpenTelemetry instance is injected into the OpenTelemetry appender. The logs in the queue are then re-emitted.

In this way (xml file and logs kept until the OpenTelemetry injection), all the logs are instrumented.

This issue aims to remove the need to declare the OpenTelemetry appender in an XML file when no optional setting is required (captureCodeAttributes, ...), the default configuration.

The OpenTelemtetry appender has to be created and configured early at the application startup to be able to instrument all the logs.

Logback provides the Configurator interface to do this. Logack uses an implementation of this interface to parse the xml files. So, a configurator implementation may be used to add the OpenTelemetry Logback appender automatically. However, Spring Boot does some reinitialisations for logging. So, not sure that a Configurator implementation would work. An alternative may be to add a new Spring logging system to automatically add the OpenTelmetry appender, in addition to the LogbackLoggingSystem. The same approach may also work for Log4j2.

[jeanbisutti]

An implementation based on a Spring logging system seems to work. I have created a draft PR: #10092 (tested with the OTel starter smoke tests).

[zeitlinger]

Great stuff - thanks for explaining how everything works together

[jeanbisutti]

It seems that only one logging system could be enabled in Spring Boot. So, the implementation of this PR would disable the LogbackLoggingSystem. I have tried to add the OpenTelemetry appender from a logging system factory and by disabling the new logging system, without success.

Another approach could be to add the OpenTelemetry appender from an application listener executed just after the LoggingApplicationListener.

[jeanbisutti]

Implemented for Logback, the default logging system in Spring Boot.

Log4j 2 does not work today with GraalVM native images. So it seems reasonable to close this issue.