Insertion failures cause segfault in concurrent index update actor

[dongxinEric]

A segmentation fault sometimes got thrown out from https://github.com/FoundationDB/fdb-document-layer/blob/master/src/QLContext.actor.cpp#L418

After some basic debugging it seems like somehow the memory of one of the item in Cartesian product iterator became invalid.

[dongxinEric]

Some observations: it seems like to happen in the following scenario:

1. When trying to insert/update multiple documents in one transaction;
2. Then one of them failed for whatever awesome reason;
3. Then other actors got canceled, theoretically;
4. But somehow some actor still were trying to fulfill their duty and thus trying to access to some memory now became invalid, and thus the seg fault.

Now magically, I cannot reproduce it locally, but we do still see seg fault happening in our nightly correctness run: about a dozen out of 100K iterations. Thanks to the solid FDB KVS backing this, and due to the fact that the actual error(step 2.) will be returned to user before the seg fault, this error will be transparent to user since the monitor process will automatically restart the main fdbdoc once it died, and there is NO correctness violation, because nothing got committed yet.

[apkar]

The batch of inserts happening in a single request is run under a single transaction with the transaction loop present in doRetry(). When an insert in this batch fails, catch block calls tr->onError on the FDB transaction. This causes the transaction to be reset. Any concurrent actors using this transaction will be left in a bad state. There are two places we need this fix.

• Transaction shouldn't reset itself to clean state on non-retryable errors. Instead, it should fail any further requests on that transaction. That gives a chance to concurrent actors to deal with cancelled transaction. This needs changes in fdbclient.
• DocTransaction should cancel all the ongoing actors in case transaction failure.

This issue deals with the second task.