From 81ef3d15bc37a85ddf5a00502ed5ca0ab60d760a Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Tue, 11 Apr 2023 11:35:19 -0600 Subject: [PATCH] Add Trace peer forwarder doc. (#3238) * Add trace peer forwarder doc. Signed-off-by: carolxob * Updated file title. Signed-off-by: carolxob * Minor updates based on tech review feedback. Signed-off-by: carolxob * Fixed header info and links. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Minor updates. Signed-off-by: carolxob * Added questions as comments to ask SME. Signed-off-by: carolxob * Added Trace analytics link. Signed-off-by: carolxob * Minor update. Signed-off-by: carolxob * Removed markdown comments and added them in GitHub. Signed-off-by: carolxob * Made updates based on tech review feedback. Signed-off-by: carolxob * Removed markdown comment. Signed-off-by: carolxob * Minor edits. Signed-off-by: carolxob * Minor change to unnecessary punctuation. Signed-off-by: carolxob * Some feedback from doc review. Signed-off-by: carolxob * Fixed broken links. Signed-off-by: carolxob * Fixed links. Signed-off-by: carolxob * Updates based on editorial feedback. Signed-off-by: carolxob * Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md Co-authored-by: Nathan Bower * Update _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md Co-authored-by: Nathan Bower * Fixed hyperlinks to include data-prepper in path for peer forwarder. Signed-off-by: carolxob --------- Signed-off-by: carolxob Co-authored-by: Nathan Bower --- .../processors/trace-peer-forwarder.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 _data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md diff --git a/_data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md b/_data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md new file mode 100644 index 0000000000..33cf1319b8 --- /dev/null +++ b/_data-prepper/pipelines/configuration/processors/trace-peer-forwarder.md @@ -0,0 +1,52 @@ +--- +layout: default +title: trace peer forwarder +parent: Processors +grand_parent: Pipelines +nav_order: 45 +--- + +# trace peer forwarder + +The `trace peer forwarder` processor is used with [peer forwarder]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/peer-forwarder/) to reduce by half the number of events forwarded in a [Trace Analytics]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/trace-analytics/) pipeline. In Trace Analytics, each event is typically duplicated when it is sent from `otel-trace-pipeline` to `raw-pipeline` and `service-map-pipeline`. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use `trace peer forwarder` to forward an event once through the `otel-trace-pipeline` instead of `raw-pipeline` and `service-map-pipeline`, which prevents unnecessary HTTP requests. + +You should use `trace peer forwarder` for Trace Analytics pipelines when you have multiple nodes. + +## Usage + +To get started with `trace peer forwarder`, first configure [peer forwarder]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/peer-forwarder/). Then create a `pipeline.yaml` file and specify `trace peer forwarder` as the processor. You can configure `peer forwarder` in your `data-prepper-config.yaml` file. For more detailed information, see [Configuring Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started/#2-configuring-data-prepper). + +See the following example `pipeline.yaml` file: + +```yaml +otel-trace-pipeline: + delay: "100" + source: + otel_trace_source: + processor: + - trace_peer_forwarder: + sink: + - pipeline: + name: "raw-pipeline" + - pipeline: + name: "service-map-pipeline" +raw-pipeline: + source: + pipeline: + name: "entry-pipeline" + processor: + - otel_trace_raw: + sink: + - opensearch: +service-map-pipeline: + delay: "100" + source: + pipeline: + name: "entry-pipeline" + processor: + - service_map_stateful: + sink: + - opensearch: +``` + +In the preceding `pipeline.yaml` file, events are forwarded in the `otel-trace-pipeline` to the target peer, and no forwarding is performed in `raw-pipeline` or `service-map-pipeline`. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice. \ No newline at end of file