Add grafana-opentelemetry-starter

[zeitlinger]

Hello,

I'm Gregor - working OpenTelemetry Java instrumentation at Grafana Labs.

The Grafana LGTM stack is a popular observability solution for OpenTelemetry, supporting logs, metrics, and tracing (available as OSS, Enterprise and Cloud).

The grafana-opentelemetry-starter bundles together

• the new micrometer tracing in Spring Boot 3 for tracing
• the existing micrometer for metrics (it would be possible to switch to micrometer OTLP when Add auto-configuration for OTLP span exporter spring-projects/spring-boot#34508 is part of Boot 3.1)
• a custom integration for logs

Well-established project with a vibrant community

The starter itself is new, but the projects that comprise the LGTM stack have vibrant communities:

• loki for logs
• mimir for metrics
• tempo for tracing
• grafana for the whole visualization, dashboards, explore, etc.

Licence

• starter: Apache 2.0
• opentelemetry-java-instrumentation: Apache 2.0
• opentelemetry-java: Apache 2.0

Issue Tracker URL

• https://github.com/grafana/grafana-opentelemetry-starter/issues

Continuous integration

• Github actions: https://github.com/grafana/grafana-opentelemetry-starter/blob/main/.github/workflows/ci.yml
• Dependabot: https://github.com/grafana/grafana-opentelemetry-starter/blob/main/.github/dependabot.yml
• Compatibility should be achieved using Spring Boot's dependency management without overriding: config

Maven Central

• Maven Central

Configuration metadata

spring-boot-configuration-processor

Version mappings

The starter supports Boot 3.0.4+

[zeitlinger]

Hi, is there anything I can do or that is missing?

[snicoll]

@zeitlinger I don't know as we haven't got the time to review the proposal yet. Sorry, and thank you for your patience.

[zeitlinger]

friendly ping 😄

[jonatan-ivanov]

My two cents/questions (sorry for the delay):

• Is this starter considered stable? From the version number, it seems so but I'm not sure I see how the -alpha versions will not break the users in the future.
• Why the snapshots repo is needed?
• Micrometer has OTLP support, I'm not sure I understand why the OTel Metrics SDK (Micrometer bridge) is needed (basically Observation API -> Micrometer Metrics API -> Registry vs. Observation API -> Micrometer Metrics API -> Micrometer Bridge (registry) -> OTel SDK -> OTel exporter), it seems to me that it introduces unnecessary complexity.

[zeitlinger]

• Is this starter considered stable? From the version number, it seems so but I'm not sure I see how the -alpha versions will not break the users in the future.

alpha is needed for internal configuration - the user visible configuration (in application.properties) is controlled by us, so we can keep it backwards compatible. I'm happy to dive into the details - but it's a longer explanation 😄

• Why the snapshots repo is needed?

This was from early local testing - can be removed if necessary

• Micrometer has OTLP support, I'm not sure I understand why the OTel Metrics SDK (Micrometer bridge) is needed (basically Observation API -> Micrometer Metrics API -> Registry vs. Observation API -> Micrometer Metrics API -> Micrometer Bridge (registry) -> OTel SDK -> OTel exporter), it seems to me that it introduces unnecessary complexity.

support for headers was missing in boot 3.0. With could create a new version targeting 3.1 to remove this complexity.
It would take away the possibility to use the otel env vars, but I'm not sure how much user value this feature.

[jonatan-ivanov]

alpha is needed for internal configuration - the user visible configuration (in application.properties) is controlled by us, so we can keep it backwards compatible. I'm happy to dive into the details - but it's a longer explanation

I'm not sure this is true for logging, users need to interact with the OTel appender, don't they? There is not layer between the users and OTel that would prevent them experiencing breaking changes, right?

This was from early local testing - can be removed if necessary

I think it would make things safer (accidentally resolving something that you don't want to) but I guess the published artifacts what matters eventually.

support for headers was missing in boot 3.0. With could create a new version targeting 3.1 to remove this complexity.

I would say support for headers was missing in Micrometer 1.10. You can use Micrometer 1.11 with Boot 3.0 (it's a drop-in replacement) and set-up headers from java (instead of a property), should still be a one-liner).

It would take away the possibility to use the otel env vars, but I'm not sure how much user value this feature.

The OTLP registry does support OTel env vars. If users report that something is missing, we can add them but for Boot users I would highly recommend using Boot properties instead of OTel env vars.

[zeitlinger]

I'm not sure this is true for logging, users need to interact with the OTel appender, don't they? There is not layer between the users and OTel that would prevent them experiencing breaking changes, right?

The main issue with logs was the stability of the SDK - which is stable now.
The appender is from the java agent repo - and therefore is still in alpha.

We could make a GrafanaAppender that simply extends OpenTelemetryAppender to guard against incompatible changes in the appender (e.g. a renamed property) - not sure if that's necessary.

support for headers was missing in boot 3.0. With could create a new version targeting 3.1 to remove this complexity.

I would say support for headers was missing in Micrometer 1.10. You can use Micrometer 1.11 with Boot 3.0 (it's a drop-in replacement) and set-up headers from java (instead of a property), should still be a one-liner).

OK, didn't know.

It would take away the possibility to use the otel env vars, but I'm not sure how much user value this feature.

The OTLP registry does support OTel env vars. If users report that something is missing, we can add them but for Boot users I would highly recommend using Boot properties instead of OTel env vars.

OK, I'll make a version that's using this approach - would it make sense to call it "3.1" to align with the boot version?

[jonatan-ivanov]

The main issue with logs was the stability of the SDK - which is stable now.
The appender is from the java agent repo - and therefore is still in alpha.

The page you linked is about the specification of the SDK (markdown file) not the implementation (code/artifacts in Maven Central). I guess what the users will interact with is the logback and log4j2 appenders (they are alpha right now). I think we should ask the Java SIG about them because maybe they are effectively stable just in the wrong place right now (moving should be ok/relatively quick I guess) and/or maybe there are plans to make the user-facing things stable. Based on the SDK spec being stable, I'm positive that the logging implementation pieces will stabilize pretty quickly.

Oh one more thing: since boot supports both logback and log4j2, I think adding a short section to the docs how to replace the OTel logback support to the OTel log4j support might be a good idea (exclude the former, include the latter).

We could make a GrafanaAppender that simply extends OpenTelemetryAppender to guard against incompatible changes in the appender (e.g. a renamed property) - not sure if that's necessary.

I think that could be a good idea but not sure if that's necessary either: let's ask the Java SIG first about the stability of the user-facing bits that the starter is depending on.

would it make sense to call it "3.1" to align with the boot version?

I think that could seem to be a good idea at first sight but following the Boot versions could be an unnecessary burden you might not want to get into. There are some starter projects that are following the Boot major version (e.g.: mybatis-spring-boot-starter), I guess mostly because they are following along since 1.x (I'm not aware if anyone would match the minor versions though). There are also starters who does not match the major versions (e.g: solace-spring-boot-starter), I think 1.x for Boot 3.x should be totally fine but I'll let others comment on that.

[zeitlinger]

The page you linked is about the specification of the SDK (markdown file) not the implementation (code/artifacts in Maven Central). I guess what the users will interact with is the logback and log4j2 appenders (they are alpha right now). I think we should ask the Java SIG about them because maybe they are effectively stable just in the wrong place right now (moving should be ok/relatively quick I guess) and/or maybe there are plans to make the user-facing things stable. Based on the SDK spec being stable, I'm positive that the logging implementation pieces will stabilize pretty quickly.

good idea - added it to the agenda next week.
I've already seen that the Java SDK in in RC and will become stable next month.

Oh one more thing: since boot supports both logback and log4j2, I think adding a short section to the docs how to replace the OTel logback support to the OTel log4j support might be a good idea (exclude the former, include the latter).

I think we can just include both since the appenders are very lightweight.

[snicoll]

This request has been opened for quite some time and we need feedback provided by @jonatan-ivanov to be processed before we can consider this again. I am going to close this now as we've discussed this internally and we'd like the integration to be more consistent with other similar entries first.