add otel content

colleenmcginnis · Aug 4, 2023 · 6a31f10 · 6a31f10
1 parent a300ad8
commit 6a31f10
Show file tree

Hide file tree

Showing 9 changed files with 668 additions and 17 deletions.
diff --git a/docs/apm-agents-opentelemetry-collect-metrics.mdx b/docs/apm-agents-opentelemetry-collect-metrics.mdx
@@ -6,4 +6,94 @@ title: Collect metrics
 tags: [ 'serverless', 'observability', 'reference' ]
 ---
 
-No source content found.
+
+{/* import TabWidgetsOpenKibanaWidget from './transclusion/tab-widgets/open-kibana-widget.mdx' */}
+
+<div id="open-telemetry-collect-metrics"></div>
+
+<DocCallOut title="Important" color="warning">
+When collecting metrics, please note that the [`DoubleValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/DoubleValueRecorder.html)
+and [`LongValueRecorder`](https://www.javadoc.io/doc/io.opentelemetry/opentelemetry-api/latest/io/opentelemetry/api/metrics/LongValueObserver.html) metrics are not yet supported.
+</DocCallOut>
+
+Here's an example of how to capture business metrics from a Java application.
+
+```java
+// initialize metric
+Meter meter = GlobalMetricsProvider.getMeter("my-frontend");
+DoubleCounter orderValueCounter = meter.doubleCounterBuilder("order_value").build();
+
+public void createOrder(HttpServletRequest request) {
+
+   // create order in the database
+   ...
+   // increment business metrics for monitoring
+   orderValueCounter.add(orderPrice);
+}
+```
+
+See the [Open Telemetry Metrics API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/api.md)
+for more information.
+
+<div id="open-telemetry-verify-metrics"></div>
+
+## Verify OpenTelemetry metrics data
+
+Use **Discover** to validate that metrics are successfully reported to ((kib)).
+
+1. Launch ((kib)):
+
+    {/* <TabWidgetsOpenKibanaWidget /> */}
+
+1. Open the main menu, then click **Discover**.
+1. Select `apm-*` as your index pattern.
+1. Filter the data to only show documents with metrics: `processor.name :"metric"`
+1. Narrow your search with a known OpenTelemetry field. For example, if you have an `order_value` field, add `order_value: *` to your search to return
+    only OpenTelemetry metrics documents.
+
+<div id="open-telemetry-visualize"></div>
+
+## Visualize in ((kib))
+
+TSVB within ((kib)) is the recommended visualization for OpenTelemetry metrics. TSVB is a time series data visualizer that allows you to use the
+((es)) aggregation framework's full power. With TSVB, you can combine an infinite number of aggregations to display complex data.
+
+{/* lint ignore ecommerce */}
+In this example eCommerce OpenTelemetry dashboard, there are four visualizations: sales, order count, product cache, and system load. The dashboard provides us with business
+KPI metrics, along with performance-related metrics.
+
+{/* ![OpenTelemetry visualizations](images/open-telemetry-collect-metrics/-ecommerce-dashboard.png) */}
+
+Let's look at how this dashboard was created, specifically the Sales USD and System load visualizations.
+
+1. Open the main menu, then click **Dashboard**.
+1. Click **Create dashboard**.
+1. Click **Save**, enter the name of your dashboard, and then click **Save** again.
+1. Let’s add a Sales USD visualization. Click **Edit**.
+1. Click **Create new** and then select **TSVB**.
+1. For the label name, enter Sales USD, and then select the following:
+
+* Aggregation: `Counter Rate`.
+* Field: `order_sum`.
+* Scale: `auto`.
+* Group by: `Everything`
+1. Click **Save**, enter Sales USD as the visualization name, and then click **Save and return**.
+1. Now let's create a visualization of load averages on the system. Click **Create new**.
+1. Select **TSVB**.
+1. Select the following:
+
+* Aggregation: `Average`.
+* Field: `system.cpu.load_average.1m`.
+* Group by: `Terms`.
+* By: `host.ip`.
+* Top: `10`.
+* Order by: `Doc Count (default)`.
+* Direction: `Descending`.
+1. Click **Save**, enter System load per host IP as the visualization name, and then click **Save and return**.
+
+    Both visualizations are now displayed on your custom dashboard.
+
+<DocCallOut title="Important" color="warning">
+By default, Discover shows data for the last 15 minutes. If you have a time-based index
+and no data displays, you might need to increase the time range.
+</DocCallOut>
diff --git a/docs/apm-agents-opentelemetry-limitations.mdx b/docs/apm-agents-opentelemetry-limitations.mdx
@@ -6,4 +6,42 @@ title: Limitations
 tags: [ 'serverless', 'observability', 'overview' ]
 ---
 
-No source content found.
+
+<div id="open-telemetry-known-limitations"></div>
+
+<div id="open-telemetry-traces-limitations"></div>
+
+## OpenTelemetry traces
+
+* Traces of applications using `messaging` semantics might be wrongly displayed as `transactions` in the APM UI, while they should be considered `spans` (see issue [#7001](https://github.com/elastic/apm-server/issues/7001)).
+* Inability to see Stack traces in spans.
+* Inability in APM views to view the "Time Spent by Span Type"  (see issue [#5747](https://github.com/elastic/apm-server/issues/5747)).
+
+<div id="open-telemetry-metrics-limitations"></div>
+
+## OpenTelemetry metrics
+
+* Inability to see host metrics in Elastic Metrics Infrastructure view when using the OpenTelemetry Collector host metrics receiver (see issue [#5310](https://github.com/elastic/apm-server/issues/5310)).
+
+<div id="open-telemetry-logs-intake"></div>
+
+## OpenTelemetry logs
+
+* <DocBadge template="technical preview" />  The OpenTelemetry logs intake via APM Server is in technical preview.
+* The application logs data stream (`app_logs`) has dynamic mapping disabled. This means the automatic detection and mapping of new fields is disabled (see issue [#9093](https://github.com/elastic/apm-server/issues/9093)).
+
+<div id="open-telemetry-otlp-limitations"></div>
+
+## OpenTelemetry Line Protocol (OTLP)
+
+APM Server supports both the [(OTLP/gRPC)]({ot_grpc}) and [(OTLP/HTTP)]({ot_http}) protocol with ProtoBuf payload.
+APM Server does not yet support JSON Encoding for OTLP/HTTP.
+
+<div id="open-telemetry-collector-exporter"></div>
+
+## OpenTelemetry Collector exporter for Elastic
+
+The [OpenTelemetry Collector exporter for Elastic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticexporter#legacy-opentelemetry-collector-exporter-for-elastic)
+was deprecated in 7.13 and replaced by the native support of the OpenTelemetry Line Protocol in
+Elastic ((observability)) (OTLP). To learn more, see
+[migration](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticexporter#migration).
diff --git a/docs/apm-agents-opentelemetry-opentelemetry-apisdk-with-elastic-apm-agents.mdx b/docs/apm-agents-opentelemetry-opentelemetry-apisdk-with-elastic-apm-agents.mdx
@@ -1,9 +1,37 @@
 ---
-id: serverlessObservabilityApmAgentsOtelOpentelemetryApisdkWithElasticApmAgents
+id: serverlessObservabilityApmAgentsOtelApisdkWithElasticApmAgents
 slug: /serverless/observability/apm-agents-opentelemetry-opentelemetry-apisdk-with-elastic-apm-agents
 title: OpenTelemetry API/SDK with Elastic APM agents
 # description: Description to be written
 tags: [ 'serverless', 'observability', 'reference' ]
 ---
 
-No source content found.
+
+<div id="open-telemetry-with-elastic"></div>
+
+Use the OpenTelemetry API/SDKs with Elastic APM agents.
+Supported Elastic APM agents translate OpenTelemetry API calls to Elastic APM API calls.
+This allows you to reuse your existing instrumentation to create Elastic APM transactions and spans.
+
+<DocCallOut title="Tip">
+If you'd like to use OpenTelemetry to send data directly to the APM server instead,
+see <DocLink id="serverlessObservabilityApmAgentsOtelNativeSupport">OpenTelemetry native support</DocLink>.
+</DocCallOut>
+
+See the relevant Elastic APM agent documentation to get started:
+
+* [Java](((apm-java-ref))/opentelemetry-bridge.html)
+* .NET (link coming soon)
+    {/* * [.NET](((apm-dotnet-ref))/opentelemetry-bridge.html) */}
+
+* [Node.js](((apm-node-ref))/opentelemetry-bridge.html)
+* [Python](((apm-py-ref))/opentelemetry-bridge.html)
+
+<div id="open-telemetry-elastic-next"></div>
+
+## Next steps
+
+* <DocLink id="serverlessObservabilityApmAgentsOtelCollectMetrics">Collect metrics</DocLink>
+* Add <DocLink id="serverlessObservabilityApmAgentsOtelResourceAttributes">Resource attributes</DocLink>
+* Learn about the <DocLink id="serverlessObservabilityApmAgentsOtelLimitations">limitations of this integration</DocLink>
+
diff --git a/docs/apm-agents-opentelemetry-opentelemetry-native-support.mdx b/docs/apm-agents-opentelemetry-opentelemetry-native-support.mdx
@@ -1,9 +1,166 @@
 ---
-id: serverlessObservabilityApmAgentsOtelOpentelemetryNativeSupport
+id: serverlessObservabilityApmAgentsOtelNativeSupport
 slug: /serverless/observability/apm-agents-opentelemetry-opentelemetry-native-support
 title: OpenTelemetry native support
 # description: Description to be written
 tags: [ 'serverless', 'observability', 'overview' ]
 ---
 
-No source content found.
+
+<div id="open-telemetry-direct"></div>
+
+The ((stack)) natively supports the OpenTelemetry protocol (OTLP).
+This means trace data and metrics collected from your applications and infrastructure can
+be sent directly to the ((stack)).
+
+* Send data to the ((stack)) from an <DocLink id="serverlessObservabilityApmAgentsOtelNativeSupport" section="send-data-from-an-opentelemetry-collector">OpenTelemetry collector</DocLink>
+* Send data to the ((stack)) from an <DocLink id="serverlessObservabilityApmAgentsOtelNativeSupport" section="send-data-from-an-opentelemetry-agent">OpenTelemetry agent</DocLink>
+
+<div id="connect-open-telemetry-collector"></div>
+
+## Send data from an OpenTelemetry collector
+
+Connect your OpenTelemetry collector instances to Elastic ((observability)) using the OTLP exporter:
+
+```yaml
+receivers:  [^1]
+  # ...
+  otlp:
+
+processors:  [^2]
+  # ...
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 2000
+  batch:
+
+exporters:
+  logging:
+    loglevel: warn  [^3]
+  otlp/elastic:  [^4]
+    # Elastic APM server https endpoint without the "https://" prefix
+    endpoint: "${ELASTIC_APM_SERVER_ENDPOINT}" <5>  [^7]
+    headers:
+      # Elastic APM Server secret token
+      Authorization: "Bearer ${ELASTIC_APM_SECRET_TOKEN}" <6>  [^7]
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [logging, otlp/elastic]
+    metrics:
+      receivers: [otlp]
+      exporters: [logging, otlp/elastic]
+    logs:  [^8]
+      receivers: [otlp]
+      exporters: [logging, otlp/elastic]
+```
+[^1]: The receivers, like the
+[OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver), that forward data emitted by APM agents, or the [host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/hostmetricsreceiver).
+[^2]: We recommend using the [Batch processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md) and the [memory limiter processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md). For more information, see [recommended processors](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md#recommended-processors).
+[^3]: The [logging exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/loggingexporter) is helpful for troubleshooting and supports various logging levels, like `debug`, `info`, `warn`, and `error`.
+[^4]: Elastic ((observability)) endpoint configuration.
+APM Server supports a ProtoBuf payload via both the OTLP protocol over gRPC transport [(OTLP/gRPC)]({ot_grpc})
+and the OTLP protocol over HTTP transport [(OTLP/HTTP)]({ot_http}).
+To learn more about these exporters, see the OpenTelemetry Collector documentation:
+[OTLP/HTTP Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) or
+[OTLP/gRPC exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter).
+[^5]: Hostname and port of the APM Server endpoint. For example, `elastic-apm-server:8200`.
+[^6]: Credential for Elastic APM {/* <DocLink id="enApmGuideSecretToken">secret token authorization</DocLink> (`Authorization: "Bearer a_secret_token"`) or <DocLink id="enApmGuideApiKey">API key authorization</DocLink> */} (`Authorization: "ApiKey an_api_key"`).
+[^7]: Environment-specific configuration parameters can be conveniently passed in as environment variables documented [here](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables) (e.g. `ELASTIC_APM_SERVER_ENDPOINT` and `ELASTIC_APM_SECRET_TOKEN`).
+[^8]:  <DocBadge template="technical preview" />  To send OpenTelemetry logs to ((stack)) version 8.0+, declare a `logs` pipeline.
+
+You're now ready to export traces and metrics from your services and applications.
+
+<DocCallOut title="Tip">
+When using the OpenTelemetry collector, you should always prefer sending data via the [`OTLP` exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) to an Elastic APM Server.
+Other methods, like using the [`elasticsearch` exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter) to send data directly to ((es)) will send data to the ((stack)),
+but will bypass all of the validation and data processing that the APM Server performs.
+In addition, your data will not be viewable in the ((kib)) ((observability)) apps if you use the `elasticsearch` exporter.
+</DocCallOut>
+
+<div id="instrument-apps-otel"></div>
+
+## Send data from an OpenTelemetry agent
+
+To export traces and metrics to APM Server, instrument your services and applications
+with the OpenTelemetry API, SDK, or both. For example, if you are a Java developer, you need to instrument your Java app with the
+[OpenTelemetry agent for Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation).
+See the [OpenTelemetry Instrumentation guides](https://opentelemetry.io/docs/instrumentation/) to download the
+OpenTelemetry Agent or SDK for your language.
+
+Define the following environment variables to configure the OpenTelemetry agent and enable communication with Elastic APM.
+
+```bash
+export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.version=1.1,deployment.environment=production
+export OTEL_EXPORTER_OTLP_ENDPOINT=https://apm_server_url:8200
+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer an_apm_secret_token"
+export OTEL_METRICS_EXPORTER="otlp" \
+export OTEL_LOGS_EXPORTER="otlp" \  [^1]
+java -javaagent:/path/to/opentelemetry-javaagent-all.jar \
+     -classpath lib/*:classes/ \
+     com.mycompany.checkout.CheckoutServiceServer
+```
+[^1]: <DocBadge template="technical preview" />  The OpenTelemetry logs intake via APM Server is currently in technical preview.
+
+<DocTable columns={[
+  {
+    "title": "",
+    "width": "50%"
+  },
+  {
+    "title": "",
+    "width": "50%"
+  }
+]}>
+  <DocRow>
+    <DocCell>`OTEL_RESOURCE_ATTRIBUTES`</DocCell>
+    <DocCell>Fields that describe the service and the environment that the service runs in. See <DocLink id="serverlessObservabilityApmAgentsOtelResourceAttributes">resource attributes</DocLink> for more information.</DocCell>
+  </DocRow>
+  <DocRow>
+    <DocCell>`OTEL_EXPORTER_OTLP_ENDPOINT`</DocCell>
+    <DocCell>APM Server URL. The host and port that APM Server listens for events on.</DocCell>
+  </DocRow>
+  <DocRow>
+    <DocCell>`OTEL_EXPORTER_OTLP_HEADERS`</DocCell>
+    <DocCell>
+      Authorization header that includes the Elastic APM Secret token or API key: `"Authorization=Bearer an_apm_secret_token"` or `"Authorization=ApiKey an_api_key"`.
+
+      For information on how to format an API key, see
+      [API keys](((apm-guide-ref))/api-key.html).
+    </DocCell>
+  </DocRow>
+  <DocRow>
+    <DocCell>`OTEL_EXPORTER_OTLP_CERTIFICATE`</DocCell>
+    <DocCell>The trusted certificate used to verify the TLS credentials of the client. (optional)</DocCell>
+  </DocRow>
+</DocTable>
+
+You are now ready to collect traces and <DocLink id="serverlessObservabilityApmAgentsOtelCollectMetrics">metrics</DocLink> before <DocLink id="serverlessObservabilityApmAgentsOtelCollectMetrics" section="verify-opentelemetry-metrics-data">verifying metrics</DocLink>
+and <DocLink id="serverlessObservabilityApmAgentsOtelCollectMetrics" section="visualize-in-((kib))">visualizing metrics</DocLink> in ((kib)).
+
+<div id="open-telemetry-proxy-apm"></div>
+
+## Proxy requests to APM Server
+
+APM Server supports both the [(OTLP/gRPC)]({ot_grpc}) and [(OTLP/HTTP)]({ot_http}) protocol on the same port as Elastic APM agent requests. For ease of setup, we recommend using OTLP/HTTP when proxying or load balancing requests to the APM Server.
+
+If you use the OTLP/gRPC protocol, requests to the APM Server must use either HTTP/2 over TLS or HTTP/2 Cleartext (H2C). No matter which protocol is used, OTLP/gRPC requests will have the header: `"Content-Type: application/grpc"`.
+
+When using a layer 7 (L7) proxy like AWS ALB, requests must be proxied in a way that ensures requests to the APM Server follow the rules outlined above. For example, with ALB you can create rules to select an alternative backend protocol based on the headers of requests coming into ALB. In this example, you'd select the gRPC protocol when the  `"Content-Type: application/grpc"` header exists on a request.
+
+For more information on how to configure an AWS ALB to support gRPC, see this AWS blog post:
+[Application Load Balancer Support for End-to-End HTTP/2 and gRPC](https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/).
+
+For more information on how APM Server services gRPC requests, see
+[Muxing gRPC and HTTP/1.1](https://github.com/elastic/apm-server/blob/main/dev_docs/otel.md#muxing-grpc-and-http11).
+
+<div id="open-telemetry-direct-next"></div>
+
+## Next steps
+
+* <DocLink id="serverlessObservabilityApmAgentsOtelCollectMetrics">Collect metrics</DocLink>
+* Add <DocLink id="serverlessObservabilityApmAgentsOtelResourceAttributes">Resource attributes</DocLink>
+* Learn about the <DocLink id="serverlessObservabilityApmAgentsOtelLimitations">limitations of this integration</DocLink>
+