Make HTTP as the default connection option for Tempo and add disclaimer for gRPC

[mathnogueira]

Describe the enhancement you'd like to see
When Tempo is deployed in a microservice infrastructure, it doesn't ship with the gRPC endpoint we use for getting a trace by ID. We are using an internal API to fetch it over gRPC.

The solution for that would be changing the default option from gRPC to HTTP in the datastore screen add a disclaimer to the gRPC option and explaining to the user that depending on how Tempo is deployed, it might not have the required endpoint Tracetest requires. They should prefer an HTTP connection instead.

[jorgeepc]

Hey @olha23 could you take a look at this ticket and help us with the disclaimer design? Thank you!

[mathnogueira]

Thomas added a very pertinent comment:

I would recommend not providing GRPC at all. While it'll appear to work in single-binary mode, that GRPC endpoint will only return traces that are still in memory. It won't pick up anything that's already written to disk. Which could cause some very difficult to debug issues for people

Basically, if tempo gets restarted, Tracetest will not be able to fetch traces. We probably need to remove the gRPC support for tempo and tell users to update the tracing backend.

I think we can add a deprecation note to it.

[jorgeepc]

Interesting @mathnogueira. What do you think @kdhamric?

[kdhamric]

Wow. Yes, we need to drop it.

By the deprecation notice, are you saying note it in the screen for a period of time, then remove the option? @mathnogueira

[olha23]

Consider removing the gRPC support for Tempo entirely i think we need provide users with clear instructions on updating their tracing backend to support HTTP connections instead and add a deprecation note to the gRPC option, maybe something like this: "Switch to HTTP for trace retrieval in Tempo! gRPC support is being phased out. Update your backend settings now to avoid interruptions. Thanks!"

[mathnogueira]

Yes @kdhamric . I think we can keep the option for a while with some kind of banner to tell users about the the deprecation and how they can migrate to HTTP.