[BUG] Encountered exception during pipeline raw-pipeline processing.

[bimlendu-plivo]

Describe the bug
The otel_trace_raw processor does not seem to work. The pipeline using this processor fails with the below
error message:

data-prepper  | 2022-09-08T07:01:33,206 [raw-pipeline-processor-worker-5-thread-1] ERROR com.amazon.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline raw-pipeline processing
data-prepper  | java.lang.ClassCastException: class io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest cannot be cast to class com.amazon.dataprepper.model.trace.Span (io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest and com.amazon.dataprepper.model.trace.Span are in unnamed module of loader 'app')
data-prepper  | 	at com.amazon.dataprepper.plugins.processor.oteltrace.OTelTraceRawProcessor.doExecute(OTelTraceRawProcessor.java:77) ~[data-prepper.jar:1.5.1]
data-prepper  | 	at com.amazon.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:55) ~[data-prepper.jar:1.5.1]
data-prepper  | 	at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:65) ~[data-prepper.jar:1.5.1]
data-prepper  | 	at com.amazon.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:55) ~[data-prepper.jar:1.5.1]
data-prepper  | 	at com.amazon.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:62) ~[data-prepper.jar:1.5.1]
data-prepper  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
data-prepper  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
data-prepper  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
data-prepper  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
data-prepper  | 	at java.lang.Thread.run(Thread.java:832) ~[?:?]

I am using the sample app in examples/trace-analytics-sample-app to test this.

I am not able to see any service or traces on dashboard. Service map does seem to work though.

To Reproduce
Steps to reproduce the behavior:

1. Follow instruction in examples/trace-analytics-sample-app.
2. Check data-prepper logs. You should see the above error.
3. Open opensearch-dashboards, and navigate to app/observability-dashboards#/trace_analytics/services.
   You should see some information in the service map section, but service and traces sections are empty.
4. In the opensearch-dashboards, check indices in Stack Management page, you should see no docs in the otel-v1-apm-span-000001 index.

Expected behavior
The example should work as expected.

Screenshots
NA

Environment (please complete the following information):

• OS: Amazon Linux
• Version: 2

Additional context
examples/trace-analytics-sample-app also dos not build unless you revert #1691.

[dlvenable]

This sample was updated to use the otel_trace_raw processor, but the otel_trace_source source is not configured to output events. It is only outputting OTel trace requests. This is likely the problem.

[graytaylor0]

Confirmed that modifying the otel_trace_source to use record_type of event fixed the issue. Have created a PR to update the configuration used for the sample app demo (#1741)

[dlvenable]

Thank you @bimlendu-plivo for reporting this issue. The PR is merged and this example should work for you now. Please let us know if you have any other issues here?

[bimlendu-plivo]

@dlvenable Thank you for the fix.

Can we also fix other examples in the repo like this one, and more importantly the plugin readme and in the opensearch docs website.

[dlvenable]

Thank you for taking note of these @bimlendu-plivo . Yes, we can include fixing those as well. I'll re-open this issue to resolve these as well.