From 071a87f816d6661c99e5dc56120bfe6bf9d3a535 Mon Sep 17 00:00:00 2001 From: David Venable Date: Fri, 9 Sep 2022 09:29:21 -0500 Subject: [PATCH 1/2] Updates Trace Analytics examples to set the record_type to event in our documentation. Also includes a short guide to migrating to Data Prepper 2.0 for Trace Analytics. Signed-off-by: David Venable --- .../otel-trace-source/README.md | 21 ++++++++++++++----- docs/trace_analytics.md | 16 ++++++++++++++ 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/data-prepper-plugins/otel-trace-source/README.md b/data-prepper-plugins/otel-trace-source/README.md index 609c012565..8beb95bd48 100644 --- a/data-prepper-plugins/otel-trace-source/README.md +++ b/data-prepper-plugins/otel-trace-source/README.md @@ -7,9 +7,16 @@ This is a source which follows the [OTLP Protocol](https://github.com/open-telem Example `.yaml` configuration: ```yaml source: - otel_trace_source: + otel_trace_source: + record_type: event ``` +### Migrating to Data Prepper 2.0 + +This documentation uses the `record_type: event` configuration which is compatible with Data Prepper 1.4 and above. For +more information see _Migrating to Data Prepper 2.0_ in the +[Trace Analytics documentation](https://github.com/opensearch-project/data-prepper/blob/main/docs/trace_analytics.md). + ## Configurations * port(Optional) => An `int` represents the port Otel trace source is running on. Default is ```21890```. @@ -21,6 +28,7 @@ source: * thread_count(Optional) => the number of threads to keep in the ScheduledThreadPool. Default is `200`. * max_connection_count(Optional) => the maximum allowed number of open connections. Default is `500`. * authentication(Optional) => An authentication configuration. By default, this runs an unauthenticated server. See below for more information. +* record_type(Optional) => A string represents the supported record data type that is written into the buffer plugin. Value options are `otlp` or `event`. Default is `otlp`. ### Authentication Configurations @@ -31,6 +39,7 @@ The following is an example of how to run the server with HTTP Basic authenticat ```yaml source: otel_trace_source: + record_type: event authentication: http_basic: username: my-user @@ -42,6 +51,7 @@ You can also explicitly disable authentication with: ```yaml source: otel_trace_source: + record_type: event authentication: unauthenticated: ``` @@ -66,10 +76,11 @@ Create the following otel-trace-source configuration in your `pipeline.yaml`. ```yaml source: otel_trace_source: - ssl: true - sslKeyCertChainFile: "/full/path/to/certfile.crt" - sslKeyFile: "/full/path/to/keyfile.key" - unframed_requests: true + record_type: event + ssl: true + sslKeyCertChainFile: "/full/path/to/certfile.crt" + sslKeyFile: "/full/path/to/keyfile.key" + unframed_requests: true ``` Generate a private key named `keyfile.key`, along with a self-signed certificate named `certfile.crt`. diff --git a/docs/trace_analytics.md b/docs/trace_analytics.md index e0f50b4d44..b9c2abde03 100644 --- a/docs/trace_analytics.md +++ b/docs/trace_analytics.md @@ -53,6 +53,7 @@ Example `otel-trace-source` with SSL and Basic Authentication enabled. Note that ```yaml source: otel_trace_source: + record_type: event ssl: true sslKeyCertChainFile: "/full/path/to/certfile.crt" sslKeyFile: "/full/path/to/keyfile.key" @@ -77,6 +78,7 @@ otel-trace-pipeline: delay: "100" source: otel_trace_source: + record_type: event ssl: false # Change this to enable encryption in transit authentication: unauthenticated: @@ -237,3 +239,17 @@ documents how to use OpenSearch Dashboards. The [Trace Tuning page](trace_tuning.md) has information to help you tune and scale Data Prepper for trace analytics use cases. + +## Migrating to Data Prepper 2.0 + +Starting in Data Prepper 1.4, the trace processing uses Data Prepper's Event model. This allows +pipeline authors the ability to configure other processors to modify spans or traces. + +To provide a migration path, Data Prepper 1.4 introduced the following changes. +* The `otel_trace_source` has an optional parameter `record_type` which can be set to `event`. When configured, it will output event objects. +* The `otel_trace_raw` replaces `otel_trace_raw_prepper` for event-based spans. +* The `otel_trace_group` replaces `otel_trace_group_prepper` for event-based spans. + +In Data Prepper 2.0, the `otel_trace_source` will only output Events. Data Prepper 2.0 also removes +`otel_trace_raw_prepper` and `otel_trace_group_prepper` entirely. To help migrate to 2.0, +you can configure your trace pipeline using the Event model. From bec4c19caedca4a08a33bcfe5bfcb08bc4f08766 Mon Sep 17 00:00:00 2001 From: David Venable Date: Thu, 6 Oct 2022 17:26:54 -0500 Subject: [PATCH 2/2] Updated the documentation for the OTel Trace Source to show how to run on either Data Prepper 2.0 or 1.x. Signed-off-by: David Venable --- data-prepper-plugins/otel-trace-source/README.md | 14 +++++++++++--- docs/trace_analytics.md | 4 ++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/data-prepper-plugins/otel-trace-source/README.md b/data-prepper-plugins/otel-trace-source/README.md index 8beb95bd48..0e16584736 100644 --- a/data-prepper-plugins/otel-trace-source/README.md +++ b/data-prepper-plugins/otel-trace-source/README.md @@ -4,7 +4,16 @@ This is a source which follows the [OTLP Protocol](https://github.com/open-telem ## Usages -Example `.yaml` configuration: +Example `.yaml` configuration for Data Prepper 2.0: + +```yaml +source: + otel_trace_source: +``` + + +Example `.yaml` configuration for Data Prepper 1.4 and above: + ```yaml source: otel_trace_source: @@ -13,8 +22,7 @@ source: ### Migrating to Data Prepper 2.0 -This documentation uses the `record_type: event` configuration which is compatible with Data Prepper 1.4 and above. For -more information see _Migrating to Data Prepper 2.0_ in the +For more information on migrating from Data Prepper 1.x to Data Prepper 2.x, see _Migrating to Data Prepper 2.0_ in the [Trace Analytics documentation](https://github.com/opensearch-project/data-prepper/blob/main/docs/trace_analytics.md). ## Configurations diff --git a/docs/trace_analytics.md b/docs/trace_analytics.md index b9c2abde03..0f57f1a61e 100644 --- a/docs/trace_analytics.md +++ b/docs/trace_analytics.md @@ -53,7 +53,7 @@ Example `otel-trace-source` with SSL and Basic Authentication enabled. Note that ```yaml source: otel_trace_source: - record_type: event + #record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0 ssl: true sslKeyCertChainFile: "/full/path/to/certfile.crt" sslKeyFile: "/full/path/to/keyfile.key" @@ -78,7 +78,7 @@ otel-trace-pipeline: delay: "100" source: otel_trace_source: - record_type: event + #record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0 ssl: false # Change this to enable encryption in transit authentication: unauthenticated: