Change crossdock end-to-end test to use "fast to start" storage

[yurishkuro]

Requirement - what kind of business use case are you trying to solve?

Make crossdock end-to-end tests reliable and fast(er).

Problem - what in Jaeger blocks you from solving the requirement?

The objective of end-to-end test is to spin up all Jaeger containers and test span collection end to end. However, collector and query service need an external shared storage, so we're using Cassandra backend, which is slow to come up and often causes tests to fail.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Use in-memory storage by implementing it as a standalone service. The Cassandra storage backend implementation is already being tested via integration separate tests, so not needed in end-to-end test.

Any open questions to address

We could use remote gRPC storage mechanism introduced in #3383.

[isaachier]

OK I believe this is a better long-term solution. Will close my PR so that this issue can be resolved instead.

[bhavin192]

I think this can be closed as #1762 is fixed.

[yurishkuro]

No, #1762 was about crossdock tests in the client repos, and they now use all-in-one. For end-to-end tests we want to use separately built components (mirroring prod deployment).

[albertteoh]

What do folks think about implementing this as a BadgerDB-backed GRPC storage plugin? I thought since we've got the logic for reading/writing spans to Badger in the Jaeger codebase, it should help reduce the effort involved.

This also unlocks a better local dev experience, not having to start cassandra/ES, which are a little heavy-weight.

[yurishkuro]

I don't think it would work since badger is still single process even via grpc plugin. What we need is a fully remote storage.

[yurishkuro]

Implementing this in #3835