Skip to content
This repository has been archived by the owner on Jun 1, 2021. It is now read-only.

Decouple replication batch size from write batch size #292

Closed
magro opened this issue Jul 17, 2016 · 2 comments
Closed

Decouple replication batch size from write batch size #292

magro opened this issue Jul 17, 2016 · 2 comments

Comments

@magro
Copy link
Contributor

magro commented Jul 17, 2016

We recently had to reduce the eventuate.log.write-batch-size (to 2 if that's relevant) to prevent "com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large" - because we're running a C* version that does not yet have the fix for CASSANDRA-10876.
As a consequence performance tests failed and we saw that event replication between locations increasingly diverged. This seems to be caused by the fact that for the replication batch size the eventuate.log.write-batch-size is used as well.

IMO the batching layer batch size (log/persistence) and the replication batch size should be independently configurable.

Right now it's fairly easy: the replication batch can be written as a batch - atomically.
If the replication batch size could be greater than log batch size several batches would have to be written. So there are probably various options to deal with failures regarding the replication acknowledgement, what currently comes to my mind:

  • If any batch write fails then reply with "failure", so that the complete replication batch is resent (assumes that events can be written more than once)
  • Kind of selective acknowledgement: acknowledge what has been written successfully (range of events), everything unconfirmed has to be resent.
@krasserm krasserm changed the title Decouple replication batch size from log batching layer batch size Decouple replication batch size from write batch size Aug 24, 2016
@krasserm krasserm removed the prio label Aug 30, 2016
@krasserm
Copy link
Contributor

krasserm commented Oct 5, 2016

@magro what's your priority for this ticket? The issues described here are addressed by #196 and #314. If it's not a blocker or high priority I'd like to close this ticket and defer support for it until work on the other two tickets start. Even if CASSANDRA-10876 has not been applied to your Cassandra version, it shouldn't harm increasing the batch size limits as explained batch write failure handling if Eventuate is the only writer to Cassandra. WDYT?

@magro
Copy link
Contributor Author

magro commented Oct 5, 2016

This is not high priority, we can close it in favor of #196.

@magro magro closed this as completed Oct 5, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants