Provide a trace_peer_forwarder plugin

[dlvenable]

Is your feature request related to a problem? Please describe.

The Core Peer Forwarding (#700) feature will replace the existing peer-forwarder. However, an ideal trace pipeline should forward events prior to the raw_trace and service_map_stateful processors.

Describe the solution you'd like

Create a trace_peer_forwarder plugin. Thus, we can create a pipeline like the following.

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
  processor:
    - trace_peer_forwarder:
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_trace_raw:
  sink:
    - opensearch:
service-map-pipeline:
  delay: "100"
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map_stateful:
  sink:
    - opensearch:

It can forward based on the traceId.

Describe alternatives you've considered (Optional)

Rely exclusively on service_map_stateful and otel_trace_raw. But, then the events will need to be forwarded twice.

[cmanning09]

I have a few questions

• One of the goals with core peer forwarder is to provide this feature natively in data prepper. I am concerned this is moving in the opposite direction. The concerns raised are valid. I think we should look to reduce forwarding events multiple times. Would it be possible to hid this from the user and only introduce a single forwarding point at the start if all aggregation processors use the same aggregation keys?
• Can we make forwarding key generic to support log aggregation pipelines which contain two sub pipeline performing an aggregation?

[dlvenable]

Would it be possible to hid this from the user and only introduce a single forwarding point at the start if all aggregation processors use the same aggregation keys?

I considered this, and I think it would make sense if the processors were in the same pipeline. However, it seems problematic to try to predict future pipelines, especially since conditional routing is on the horizon.

Can we make forwarding key generic to support log aggregation pipelines which contain two sub pipeline performing an aggregation?

I'd like to suggest that we have a peer-forwarder: plugin which can perform peer-forwarding on user-defined keys. I suggested that we create the trace-peer-forwarder specifically to ease the burden for a user using trace analytics since it will define the key.

One of the goals with core peer forwarder is to provide this feature natively in data prepper.

I agree with this as well. But, to be clear, either of these possible plugins (trace-peer-forwarder or peer-forwarder) would use the core peer forwarding capabilities.