Improve testing

[jackshirazi]

This

• Adds a local Elasticsearch instance
• Starts the elastic agent as an otel collector with the traces configuration, bound to 0.0.0.0:4318
• Adds a collector.banana endpoint pointing at localhost:4318 into the cluster to let the agents connect to the collector
• Changes the python, nodejs and java configurations/images to generate traces from the app
• Confirms that the traces have propagated through to Elasticsearch

So the test is now a full end-to-end trace test. The test is also setup to switch to run on both amd64 and arc64, though the arc64 option is commented for the moment

[rogercoll]

Could you provide more information about why this Collector instance is needed? Why not using the Operator's endpoint?

[jackshirazi]

The flow of trace data is app->app agent->collector->ES, so it needs a collector. The operator endpoint is for mutating the k8s definitions so that's entirely separate, that's not involved in the flow of traces

[rogercoll]

I might be missing something, but if you are using the values.yaml file we ship to deploy the Operator, it will deploy a collector in each K8s node with an open OTLP endpoint for traces. Those collectors will have the same processing components that the one you are launching manually.

[jackshirazi]

ah gotcha. yes that didn't seem to be working in my tests. It was easier for me to spin up a new collector with the traces config. If you think it's straightforward to have it running so that it sends traces, we can do that, I agree it would be better

[swiatekm]

If the data already has the right shape, then your collector here could just receive data and forward it to ES, if you really want to have it. It doesn't need to use this full config.

Though if it was up to me, I'd just ES itself in the K8s cluster and forward the 9200 port.

[jackshirazi]

The collector does some transforms on the incoming data from the application agents, so I think it needs to be correctly configured for handling incoming trace data.