Empty span name list in UI with Cassandra3 storage component

[llinder]

While testing the Cassandra3 storage component I noticed that the span names dropdown in the UI went from showing 10+ entries to 0. Running the same CQL statement through cqlsh shows that there are over 1000 records with blank span names. This appears to be causing the UI to no longer show any span names after a period of time.

/cc @michaelsembwever

[michaelsembwever]

Span names in the Cassandra3 storage are stored within the trace_by_service_span table. The manual denormalisation of the span name list to a separate table no longer exists, instead it is implicitly denormalised via a materialized view to the trace_by_service table.

The UI calls store.getSpanNames(serviceName) for a given serviceName and this does the cql query: SELECT span_name FROM trace_by_service where service_name = '<service_name>' LIMIT 100;

The materialized view doesn't actually need to store blank span_names (the origin table trace_by_service_span does though). Unfortunately materialized views can not yet filter out non-blank strings yet, they can only filter out the non-nulls.

The only approach valid that I can see is to filter out blank span_names at runtime time in the cassandraSpanStore.getSpanNames(serviceName) method. This is a shitty solution, but I suspect it'll be better solved in future C* versions.

(( It might also be worth investigating performance differences when the bucket cluster key in the materialized view is defined after the span_name clustering key. ))

[llinder]

Another problem with the materialized view is that it doesn't return a distinct list of span names. In other words it would have the same problem with non blank span names in that it will only return the first N which might not include all of the non blank names either.

My quick solution was to just create another table to hold service_name and span_name like the Cassandra2 storage component. This kind of defeats the purpose of this experimental storage implementation but at least it still enables duration and text search on annotations.

[michaelsembwever]

Another problem with the materialized view is that it doesn't return a distinct list of span names. In other words it would have the same problem with non blank span names in that it will only return the first N which might not include all of the non blank names either.

That's correct. The suggestion was to ensure you included enough non-blank names.

The manual denormalisation over a materialized view works around the limitation nicely though, will check it out.