update open telemetry docs for telemetry 1.1 #7033
Conversation
| In microservice applications, sources of latency or inaccuracy can be difficult to determine because relationships and dependencies among the constituent services are not always obvious. MicroProfile Telemetry helps you collect data on the paths that application requests take through services. | ||
|
|
||
| One way to increase observability of an application is by emitting traces. Traces represent requests and consist of multiple spans. A span represents a single operation in a request and contains a name, time-related data, log messages, and metadata to gather data about what occurs during a transaction. | ||
|
|
||
| link:https://projects.eclipse.org/projects/technology.microprofile/releases/microprofile-telemetry-1.0/plan%E2%80%A8%E2%80%A822.0.0.10-bet[MicroProfile Telemetry] is based on the https://opentelemetry.io/[OpenTelemetry project], which is a collection of open source vendor-agnostic tools, APIs, and SDKs for creating and managing trace data. You can enable MicroProfile Telemetry for Open Liberty by adding the feature:mpTelemetry[display=MicroProfile Telemetry] feature to your `server.xml` file and configuring the feature to connect to your distributed trace service. | ||
|
|
||
| MicroProfile Telemetry supports the instrumentation of tracing data in your code in three different ways: <<#auto, automatic instrumentation>>, <<#manual, manual instrumentation>>, or <<#agent, agent instrumentation>>. The data can then be exported to tracing systems like https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. |
There was a problem hiding this comment.
This section was about how the user configures their liberty server, in the middle of paragraphs about the background of the open telemetry project. Moved to its own section.
| MicroProfile Telemetry replaces MicroProfile OpenTracing. For information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see xref:reference:diff/mp-50-60-diff.adoc#telemetry[Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0]. | ||
|
|
||
| == Using OpenTelemetry == |
There was a problem hiding this comment.
This section has a high level overview of the three ways of using open telemetry and examples of the configuration that is common to all three.
Those examples are intentionally duplicated in the individual sections. This way if you read the page top down you get a nice landing spot, and if you jump to one section and skip everything else you don't miss anything.
There was a problem hiding this comment.
Should this be Using MicroProfile Telemetry? given we are Wrapping Otel and doing things via the MP way
There was a problem hiding this comment.
Given more thought, should this be Using OpenTelemetry with MicroProfile Telemetry as just saying basically OTEL is confusing when we start refering to MP Telemetry
benjamin-confino
force-pushed
the
vNext
branch
2 times, most recently
from
276a638
to
a715f30
Compare
| [#auto] | ||
| == Automatic instrumentation | ||
| With automatic instrumentation, you can observe traces without modifying the source code in your Jakarta RESTful web service (formerly JAX-RS) applications. To start emitting traces with automatic instrumentation, enable the feature:mpTelemetry[display=MicroProfile Telemetry] feature in your `server.xml` file. By default, MicroProfile Telemetry tracing is off. To enable tracing, specify the `otel.sdk.disabled=false` MicroProfile Config property and any exporter configuration that your tracing service service requires. | ||
| With automatic instrumentation, you can observe traces without modifying the source code in your JAX-RS or Jakarta RESTful web service applications. To start emitting traces with automatic instrumentation, enable the feature:mpTelemetry[display=MicroProfile Telemetry] feature in your `server.xml` file by adding `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. By default, MicroProfile Telemetry automatic instrumentation is off. To enable tracing, specify the `otel.sdk.disabled=false` MicroProfile Config property and any exporter configuration that your tracing service service requires. |
There was a problem hiding this comment.
Looking at the FATs, you can use manual instrumentation without otel.sdk.disabled=false so replace the word tracing above.
TODO: Double check this.
There was a problem hiding this comment.
I don't think that's true, which FAT test were you looking at?
There was a problem hiding this comment.
I went looking for counter examples in the publish/server. The string "sdk" does not appear inside Telemetry10Bean
There was a problem hiding this comment.
Telemetry10Bean is used by TelemetryBeanTest which sets this property in the config file within the application: https://github.com/OpenLiberty/open-liberty/blob/062e708ba9dbf2db640fc4c5f3f2dc4ec54dd5f9/dev/io.openliberty.microprofile.telemetry.1.0.internal_fat/fat/src/io/openliberty/microprofile/telemetry/internal_fat/TelemetryBeanTest.java#L57
There was a problem hiding this comment.
Better turn automatic instrumentation back to tracing
| @@ -79,6 +112,15 @@ Automatic instrumentation is available only for Jakarta RESTful web service appl | |||
| </dependency> | |||
| ---- | |||
|
|
|||
| - Configure an exporter. For example, to export traces to a Jaeger server with the OpenTelemetry Protocol (OTLP) enabled, add the following entries to your `bootstrap.properties` file. | |||
There was a problem hiding this comment.
This section is already long so I think we can forgo an example of zipkin here
benjamin-confino
force-pushed
the
vNext
branch
2 times, most recently
from
fe84f4f
to
31618bb
Compare
| To enable OpenTelemetry you must add one the following features `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. Then you configure one or more of: | ||
|
|
||
| * <<#auto, automatic instrumentation>> - which automatically traces your JAX-RS or RESTful web services. | ||
| * <<#manual, manual instrumentation>> - which allows you to start and end telemetry spans manually. | ||
| * <<#agent, agent instrumentation>> - which automatically adds telemetry to popular open source libraries. Agent instrumentation imposes additional limitations which are documented in <<#agent, agent instrumentation>>. |
There was a problem hiding this comment.
I don't think this is quite true.
While you do have to configure the mpTelemetry features in order for them to do anything, you don't configure manual and automatic instrumentation separately.
I don't think we should mention agent instrumentation here at all. The agent is provided by the OpenTelemetry project and we don't provide any support for it (other than testing to ensure openliberty works nicely with it). Whenever we mention instrumentation using the agent, it should be clear that the agent is a separate tool provided by another group, not part of Open Liberty and not supported by us.
There was a problem hiding this comment.
I think you do. For automatic you add otel.sdk.disabled=false. For manual you add third party to the classloader.
I'd lean towards mentioning agent instrumentation. Its hard enough to find open telemetry on the web as it is, but willing to refer to you and Emily on that.
There was a problem hiding this comment.
I'll change configure to use given the discussion about sdk.disabled.
benjamin-confino
force-pushed
the
vNext
branch
from
31618bb
to
5a3c96f
Compare
| MicroProfile Telemetry replaces MicroProfile OpenTracing. For information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see xref:reference:diff/mp-50-60-diff.adoc#telemetry[Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0]. | ||
|
|
||
| == Using OpenTelemetry == | ||
|
|
||
| To enable OpenTelemetry you must add one the following features `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. |
There was a problem hiding this comment.
So to clarify here. Even if you enable either of the otel features, you need to set otel.sdk.disabled=false.
There was a problem hiding this comment.
Should we also mention here that corresponding mp umbrella features include mptelemetry. looking at the feature files it is just 6.0(1.0) and 6.1(1.1) as the older ones we are supporting include OpenTracing.
A section on updating from OT to OTEL might be useful to have as well if we (I am asusming here) that we want EE7/EE8/EE9 customers to use OTEL instead of OT
|
|
||
| To enable OpenTelemetry you must add one the following features `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. | ||
|
|
||
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: |
There was a problem hiding this comment.
This looks like two pieces smashed together. I would split into Disabled/enabled requirement and then the backend config. I am ok with reason you have given for the combined examples as I could see someone forgetting to set the enablement flag when they configure their exporter
By Default OpenTelemetry is disabled (with explanation as to why maybe). To enable the generation of traces, set the property `otel.sdk.disabled=false`.
To store the generated traces a OpenTelemetry storage solution such as Jaeger or Zipkin is required. To export the generated traces to these, an exporter definition is required that includes the exporter type and the endpoint to be connected to. For example:
There was a problem hiding this comment.
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: | |
| By default tracing is disabled so you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: |
A back end isn't strictly required. You can export to logs for instance
|
|
||
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: | ||
|
|
||
| Configuring OpenTelemetry to use a Jager server: |
There was a problem hiding this comment.
Which file do these properties get set in, later you mention bootstrap.properties, but given the most recent file mentioned to this point is 'server.xml' that is currently where I would put these
There was a problem hiding this comment.
Thanks Alex, I implemented both your changes
|
|
||
| === Telemetry Versions and Java Enterprise Editions === | ||
|
|
||
| * `mpTelemetry-1.1` is compatible with `jakartaee-10.0` and `microProfile-6.0` |
There was a problem hiding this comment.
This is backwards. 1.0 supports JEE10 and MP6.0 only, 1.1 supports JEE 7, 8, 9 and 10 with MP 1.4, 4.1. 5.0 and 6.1 respectively(slide 10 of the UFO https://ibm.ent.box.com/file/1266959749307?s=7lqz6kpmyole7k1bp9kih85789ma4y8x)
| [#auto] | ||
| == Automatic instrumentation | ||
| With automatic instrumentation, you can observe traces without modifying the source code in your Jakarta RESTful web service (formerly JAX-RS) applications. To start emitting traces with automatic instrumentation, enable the feature:mpTelemetry[display=MicroProfile Telemetry] feature in your `server.xml` file. By default, MicroProfile Telemetry tracing is off. To enable tracing, specify the `otel.sdk.disabled=false` MicroProfile Config property and any exporter configuration that your tracing service service requires. | ||
| With automatic instrumentation, you can observe traces without modifying the source code in your JAX-RS or Jakarta RESTful web service applications. To start emitting traces with automatic instrumentation, enable the feature:mpTelemetry[display=MicroProfile Telemetry] feature in your `server.xml` file by adding `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. By default, MicroProfile Telemetry tracing is off. To enable tracing, specify the `otel.sdk.disabled=false` MicroProfile Config property and any exporter configuration that your tracing service service requires. | ||
|
|
||
| For example, to export traces to a Jaeger server with the OpenTelemetry Protocol (OTLP) enabled, add the following entries to your `bootstrap.properties` file. | ||
|
|
There was a problem hiding this comment.
The examples that follow this comments in the doc, but as they are unchanged I can comment on them are now a duplicate of the examples above. and the otlp example should go with the others
There was a problem hiding this comment.
This is intentional. The section Using OpenTelemetry was written to cover the common settings needed to configure OpenTelemetry and summarise the other sections so the reader knows where to go.
However the sections with specific ways to use OpenTelemetry were written so they'd be complete if a reader says "I know I need automatic telemetry so I'll skip to that section" and ignores everything else.
I think duplicate examples is a price worth paying for having both an overview section and fully-self contained specific sections. A bit of duplication now saves a PMR tomorrow
There was a problem hiding this comment.
it is a small amount of duplication and as you say. the so called Expert user will jump to the section rather then read the rest.
However, we give references for Auto and Manual for these configurations, but not Agent, are they needed for the Agent as I am free that Agent doesn't need the flag or the exporter properties defined in the server.xml
| @@ -50,7 +86,16 @@ For more information about the available properties, see xref:microprofile-confi | |||
|
|
|||
| [#manual] | |||
| == Manual instrumentation | |||
| Automatic instrumentation is available only for Jakarta RESTful web service applications. To create spans for other operations, such as database calls, you can add manual instrumentation to the source code for those operations by using the https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/1.19.0/io/opentelemetry/api/trace/package-summary.html[OpenTelemetry API]. However, before you manually instrument your code, you must complete the following prerequisites. | |||
| Automatic instrumentation is available only for JAX-RS and Jakarta RESTful web service applications. To create spans for other operations, such as database calls, you can add manual instrumentation to the source code for those operations by using the https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/1.19.0/io/opentelemetry/api/trace/package-summary.html[OpenTelemetry API]. However, before you manually instrument your code, you must complete the following prerequisites. | |||
There was a problem hiding this comment.
Is reference to Automatic needed here, given it is literally the section above and covers the areas it can be automatically done, as I assume we can manually instrument Jax-rs if we wanted to.
|
|
||
| - Enable OpenTelemetry by setting `otel.sdk.disabled=false` and configure an exporter. For example, to export traces to a Jaeger server with the OpenTelemetry Protocol (OTLP) enabled, add the following entries to your `bootstrap.properties` file. | ||
|
|
||
| [source,properties] |
There was a problem hiding this comment.
repeat of existing definitions which now feels orphaned
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: | ||
|
|
||
| Configuring OpenTelemetry to use a Jager server: | ||
| [source,properties] |
There was a problem hiding this comment.
When previewing in github I am seeing some strange rendering that will need fixing. here is a portion of the page as it is being rendered
benjamin-confino
force-pushed
the
vNext
branch
from
5a3c96f
to
02e524d
Compare
| @@ -12,19 +12,55 @@ | |||
| :page-type: general | |||
| = Enable distributed tracing with MicroProfile Telemetry | |||
|
|
|||
| == What is OpenTelemetry == | |||
There was a problem hiding this comment.
| == What is OpenTelemetry == | |
| == Introducing OpenTelemetry == |
There was a problem hiding this comment.
Having a question as the title doesn't seem right to me
|
|
||
| To enable OpenTelemetry you must add one the following features `<feature>mpTelemetry-1.0</feature>` or `<feature>mpTelemetry-1.1</feature>` to your server.xml file. | ||
|
|
||
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: |
There was a problem hiding this comment.
| By default OpenTelemetry is disabled, and OpenTelemetry requires a telemetry back end such as https://www.jaegertracing.io/[Jaeger] or https://zipkin.io/[Zipkin]. So you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: | |
| By default tracing is disabled so you have to set the property `otel.sdk.disabled=false` to enable OpenTelemetry and configure a backend. For example: |
A back end isn't strictly required. You can export to logs for instance
benjamin-confino
force-pushed
the
vNext
branch
from
02e524d
to
189248e
Compare
benjamin-confino
force-pushed
the
vNext
branch
from
189248e
to
b38cb47
Compare
| ---- | ||
| otel.sdk.disabled=false | ||
| otel.traces.exporter=logging | ||
| otel.exporter.zipkin.endpoint=http://localhost:9411/api/v2/spans |
There was a problem hiding this comment.
Remove this line. You would not need the Zipkin exporter endpoint if you are exporting to logs
benjamin-confino
force-pushed
the
vNext
branch
from
b38cb47
to
3f6eca5
Compare
benjamin-confino
force-pushed
the
vNext
branch
from
3f6eca5
to
447d02d
Compare
|
Hi @benjamin-confino - I see you resolved conflicts by merging draft into your vNext fork. I think this is going to cause branch conflicts when this update is pushed to staging and vNext, as draft is not in sync with those branches. I think the easiest way to fix this is, once your changes are ready to merge, copy your updated file into the microprofile-telemetry-troubleshooting branch that Yasmin created, since that is a prod based branch that is currently in sync with draft. Our contribution process is documented here, which describes the draft->staging->vNext publishing flow. Though this is an edge case where two writers were working at the same time in the same file from different branches. Sorry for the inconvenience. |
benjamin-confino
dismissed
yasmin-aumeeruddy’s stale review
The base branch was changed.
benjamin-confino
changed the base branch from
staging
to
microprofile-telemetry-troubleshooting
benjamin-confino
force-pushed
the
vNext
branch
from
ee9c15d
to
447d02d
Compare
Can do |
yasmin-aumeeruddy
merged commit ac5b989
into
OpenLiberty:microprofile-telemetry-troubleshooting
No description provided.