Avoid exception in load-tester when adding then removing node in the …

…cluster.

Summary: Curently, when adding then removing node in a cluster while a Cassandra load-tester is run, "TransportException: Connection has been closed" is reported. This revision improves the load-tester by enabling automatic retry of the statements by setting statements as idempotent by default. Logging is also added to the default retry policy.

Test Plan:
```
$ yb-ctl start
$ yb-ctl add_node
$ yb-ctl remove_node 4

2018-05-04 13:43:13,170 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Using a randomly generated UUID : ee80ecb7-8d4d-4736-bf7c-ef95ac70ccb4
2018-05-04 13:43:13,178 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] App: CassandraKeyValue
2018-05-04 13:43:13,178 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Run time (seconds): -1
2018-05-04 13:43:13,178 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Adding node: 127.0.0.1:9042
2018-05-04 13:43:13,180 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Num reader threads: 24, num writer threads: 2
2018-05-04 13:43:13,180 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Num unique keys to insert: 1000000
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Num keys to update: -1000001
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Num keys to read: -1
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Value size: 0
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Restrict values to ASCII strings: false
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Perform sanity check at end of app run: false
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Table TTL (secs): -1
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Local reads: false
2018-05-04 13:43:13,181 [INFO|com.yugabyte.sample.common.CmdLineOpts|CmdLineOpts] Read only load: false
2018-05-04 13:43:13,354 [INFO|com.yugabyte.sample.apps.AppBase|AppBase] Connecting to nodes: /127.0.0.1:9042
2018-05-04 13:43:13,651 [WARN|com.datastax.driver.core.NettyUtil|NettyUtil] Found Netty's native epoll transport, but not running on linux-based operating system. Using NIO instead.
2018-05-04 13:43:17,367 [INFO|com.yugabyte.sample.apps.AppBase|AppBase] Created a Cassandra table using query: [CREATE TABLE IF NOT EXISTS CassandraKeyValue (k varchar, v blob, primary key (k));]
2018-05-04 13:43:22,371 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 5937.96 ops/sec (3.80 ms/op), 29697 total ops  |  Write: 290.25 ops/sec (6.86 ms/op), 1452 total ops  |  Uptime: 5004 ms | maxWrittenKey: 1452 | maxGeneratedKey: 1454 |
2018-05-04 13:43:27,372 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7032.53 ops/sec (3.41 ms/op), 64878 total ops  |  Write: 322.90 ops/sec (6.18 ms/op), 3067 total ops  |  Uptime: 10005 ms | maxWrittenKey: 3065 | maxGeneratedKey: 3068 |
2018-05-04 13:43:32,375 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7881.90 ops/sec (3.04 ms/op), 104316 total ops  |  Write: 337.56 ops/sec (5.94 ms/op), 4756 total ops  |  Uptime: 15008 ms | maxWrittenKey: 4755 | maxGeneratedKey: 4757 |
2018-05-04 13:43:37,377 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7621.39 ops/sec (3.07 ms/op), 142437 total ops  |  Write: 328.28 ops/sec (5.98 ms/op), 6398 total ops  |  Uptime: 20010 ms | maxWrittenKey: 6380 | maxGeneratedKey: 6399 |
2018-05-04 13:43:42,381 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 3103.47 ops/sec (7.56 ms/op), 157968 total ops  |  Write: 133.48 ops/sec (14.55 ms/op), 7066 total ops  |  Uptime: 25014 ms | maxWrittenKey: 7065 | maxGeneratedKey: 7067 |
2018-05-04 13:43:47,384 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 3125.77 ops/sec (7.96 ms/op), 173605 total ops  |  Write: 145.12 ops/sec (14.28 ms/op), 7792 total ops  |  Uptime: 30017 ms | maxWrittenKey: 7785 | maxGeneratedKey: 7793 |
2018-05-04 13:43:52,387 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 3071.95 ops/sec (7.86 ms/op), 188974 total ops  |  Write: 127.53 ops/sec (15.80 ms/op), 8430 total ops  |  Uptime: 35020 ms | maxWrittenKey: 8428 | maxGeneratedKey: 8431 |
2018-05-04 13:43:57,388 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 1407.73 ops/sec (13.10 ms/op), 196015 total ops  |  Write: 60.18 ops/sec (25.59 ms/op), 8731 total ops  |  Uptime: 40021 ms | maxWrittenKey: 8727 | maxGeneratedKey: 8732 |
2018-05-04 13:44:02,390 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 4248.71 ops/sec (6.98 ms/op), 217266 total ops  |  Write: 199.53 ops/sec (12.35 ms/op), 9729 total ops  |  Uptime: 45023 ms | maxWrittenKey: 9728 | maxGeneratedKey: 9730 |
2018-05-04 13:44:07,392 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7329.11 ops/sec (3.27 ms/op), 253933 total ops  |  Write: 319.81 ops/sec (6.25 ms/op), 11329 total ops  |  Uptime: 50025 ms | maxWrittenKey: 11328 | maxGeneratedKey: 11330 |
2018-05-04 13:44:12,396 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7865.44 ops/sec (3.05 ms/op), 293287 total ops  |  Write: 331.17 ops/sec (6.04 ms/op), 12986 total ops  |  Uptime: 55029 ms | maxWrittenKey: 12985 | maxGeneratedKey: 12987 |
2018-05-04 13:44:17,396 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7876.12 ops/sec (3.05 ms/op), 332673 total ops  |  Write: 342.55 ops/sec (5.84 ms/op), 14699 total ops  |  Uptime: 60029 ms | maxWrittenKey: 14698 | maxGeneratedKey: 14700 |
2018-05-04 13:44:22,397 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 2529.45 ops/sec (3.19 ms/op), 345322 total ops  |  Write: 112.58 ops/sec (5.96 ms/op), 15262 total ops  |  Uptime: 65030 ms | maxWrittenKey: 15260 | maxGeneratedKey: 15263 |
2018-05-04 13:44:27,397 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 499.93 ops/sec (79.89 ms/op), 347822 total ops  |  Write: 15.40 ops/sec (189.09 ms/op), 15339 total ops  |  Uptime: 70030 ms | maxWrittenKey: 15263 | maxGeneratedKey: 15340 |
2018-05-04 13:44:32,402 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7731.20 ops/sec (3.10 ms/op), 386518 total ops  |  Write: 349.04 ops/sec (6.92 ms/op), 17086 total ops  |  Uptime: 75035 ms | maxWrittenKey: 17085 | maxGeneratedKey: 17087 |
2018-05-04 13:44:37,404 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7470.70 ops/sec (3.21 ms/op), 423882 total ops  |  Write: 411.28 ops/sec (4.86 ms/op), 19143 total ops  |  Uptime: 80037 ms | maxWrittenKey: 19142 | maxGeneratedKey: 19144 |
2018-05-04 13:44:42,404 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7551.54 ops/sec (3.18 ms/op), 461646 total ops  |  Write: 412.33 ops/sec (4.85 ms/op), 21205 total ops  |  Uptime: 85037 ms | maxWrittenKey: 21204 | maxGeneratedKey: 21206 |
2018-05-04 13:44:47,408 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 7336.85 ops/sec (3.27 ms/op), 498356 total ops  |  Write: 365.54 ops/sec (5.47 ms/op), 23034 total ops  |  Uptime: 90041 ms | maxWrittenKey: 23033 | maxGeneratedKey: 23035 |
```

Reviewers: bharat

Reviewed By: bharat

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D4751

java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/AppBase.java

@@ -34,9 +34,12 @@
 
 import com.datastax.driver.core.Cluster;
 import com.datastax.driver.core.Session;
+import com.datastax.driver.core.QueryOptions;
 import com.datastax.driver.core.SimpleStatement;
 import com.datastax.driver.core.exceptions.NoHostAvailableException;
 import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
+import com.datastax.driver.core.policies.DefaultRetryPolicy;
+import com.datastax.driver.core.policies.LoggingRetryPolicy;
 import com.yugabyte.driver.core.policies.PartitionAwarePolicy;
 import com.yugabyte.sample.common.CmdLineOpts;
 import com.yugabyte.sample.common.CmdLineOpts.ContactPoint;
@@ -162,7 +165,10 @@ protected static synchronized void createCassandraClient(List<ContactPoint> cont
         builder.withLoadBalancingPolicy(
           new PartitionAwarePolicy());
       }
-      cassandra_cluster = builder.build();
+      cassandra_cluster =
+          builder.withQueryOptions(new QueryOptions().setDefaultIdempotence(true))
+                 .withRetryPolicy(new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE))
+                 .build();
       LOG.debug("Connected to cluster: " + cassandra_cluster.getClusterName());
     }
     if (cassandra_session == null) {