Cassandra driver's connections per host option

[drax68]

Please add configurable option to manage maxConnectionPerHost cassandra driver parameter.
Defaults is quite low (2 for HostDistance.REMOTE) and causes multiple errors in collector logs.

[codefromthecrypt]

@michaelsembwever @yurishkuro @prat0318 @danchia any reason we shouldn't expose CASSANDRA_MAX_CONNECTIONS?

[danchia]

I can't think of any reason why you wouldn't want to, it's quite a reasonable tuning option.

Speaking of which, one improvement I had planned to test was batching writes to Cassandra into small, unlogged batches, which I think would improvement performance.

[prat0318]

++ would be nice

[yurishkuro]

batching writes to Cassandra into small, unlogged batches, which I think would improvement performance.

I've been told quite the opposite by the team with more experience with Cassandra. A batch can span multiple nodes, so batching puts more load on the coordinating node, rather than that work to be done by a client with token-aware connection.

So you may want to profile it first, before changing the storage implementation to support batches.

[codefromthecrypt]

Here's an impl #958

[danchia]

I was basing it off: https://dzone.com/articles/efficient-cassandra-write, which could mean there is some benefit to be had if we have locality.

I think the other thing that was motivating me to batch was that AFAIK we wait for the each trace to be stored before processing the next one, at least for Kafka. This greatly reduces potential throughput than say, if we allowed for up to X number of store operations in flight in the collector.

[yurishkuro]

Thanks for the link, Daniel. It seems the articles agrees with what I said, that fan-out from the client is more efficient then batching and sending to a coordination node. But if collector reads KAFKA and writes to C one span at a time, then yes it would be bad and batching might help. It's just you'd be optimizing the wrong end. We don't use collectors over KAFKA yet (actually working on it for another use case), but our collectors (in Go) have an internal queue which is feeding N go-routines, over a token-aware connection. The same thing can be easily done in Scala/Java collectors.

Also worth noting that Cassandra connections are multiplexing, I think up to 128 or 256 streams, but we haven't gone that high in the # of threads.

[codefromthecrypt]

@danchia @yurishkuro booting the batch discussion to #961 where it is easier for others to identify

[codefromthecrypt]

fix pending on whether we want two properties (for local and remote), as this issue was brought up about remote.

[codefromthecrypt]

I wonder if we addressed this, if we'd end up with only needing to tune the local policy? #967

[codefromthecrypt]

here's a PR which allows non-local hosts to be skipped #980