multiple spans and kafka collectors

[codefromthecrypt]

SpanStore.apply is invoked by collectors to store spans in bulk (even if not in the same trace). The kafka receiver only sends a single span to this method. That means if optimizations are made on span stores (such as #961 for cassandra), kafka users won't necessarily benefit.

Options to consider

The kafka transport encodes a span in a single KV message (key is null) at the moment. Let's see if there's a way to leverage the bulk storage api: SpanStore.apply.

Send multiple spans per message and adjust kafka decoding

Kafka messages are currently single-span, and a single-entry list is not the same thrift bytes as a single-entry. To support multiple-span messages, we need to either peek at the bytes or vary on message key (adjusting collectors accordingly)

separate out reading off kafka from writing to the span store

@yurishkuro mentions that reading from kafka into a local queue might help. For example, a span store reader could consume N spans off that queue. #961 (comment)

How other transports work

Scribe

The scribe transport encodes a span in a single log entry. However the scribe api allows you to log multiple entries, which means that the scribe receiver can handle a bunch of spans at the same time (same network request).

Http

The http transport encodes a list of spans in json or thrift format, these are passed directly to the span store.

TChannel

TChannel-based communication between clients and central collectors batch multiple spans into one request.

[codefromthecrypt]

from @prat0318

i would go with Send multiple spans per message option just because of simplicity and not adding another point of failure (local queue)