Fix race in clear scroll

[jasontedor]

Here is the problem: if two threads are racing and one hits a failure freeing a context and the other succeeded, we can expose the value of the has failure marker to the succeeding thread before the failing thread has had a chance to set the failure marker. This is a problem if the failing thread counted down the expected number of operations, then be put to sleep by a gentle lullaby from the OS, and then the other thread could count down to zero. Since the failing thread did not get to set the failure marker, the succeeding thread would respond that the clear scroll succeeded and that makes that thread a liar. This commit addresses by first setting the failure marker before we potentially expose its value to another thread.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jasontedor]

This addresses this build failure:

17:06:14    2> jun 11, 2018 3:06:14 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
17:06:14    2> ADVERTENCIA: Uncaught exception in thread: Thread[Thread-92,5,TGRP-ClearScrollControllerTests]
17:06:14  Suite: org.elasticsearch.action.search.ClearScrollControllerTests
17:06:14    2> java.lang.AssertionError
17:06:14    1> [2018-06-11T15:06:14,077][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearScrollIdsWithFailure]: before test
17:06:14    2> 	at __randomizedtesting.SeedInfo.seed([B475D8195532593A]:0)
17:06:14    2> 	at org.junit.Assert.fail(Assert.java:86)
17:06:14    1> [2018-06-11T15:06:14,085][WARN ][o.e.a.s.ClearScrollControllerTests] Clear SC failed on node[{node_2}{x49Meh1-TFaqWs_0u2chCQ}{0.0.0.0}{0.0.0.0:2}]
17:06:14    2> 	at org.junit.Assert.assertTrue(Assert.java:41)
17:06:14    2> 	at org.junit.Assert.assertFalse(Assert.java:64)
17:06:14    2> 	at org.junit.Assert.assertFalse(Assert.java:74)
17:06:14    1> java.lang.IllegalArgumentException: boom
17:06:14    2> 	at org.elasticsearch.action.search.ClearScrollControllerTests$5.onResponse(ClearScrollControllerTests.java:195)
17:06:14    2> 	at org.elasticsearch.action.search.ClearScrollControllerTests$5.onResponse(ClearScrollControllerTests.java:189)
17:06:14    2> 	at org.elasticsearch.action.search.ClearScrollController.onFreedContext(ClearScrollController.java:130)
17:06:14    2> 	at org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$2(ClearScrollController.java:115)
17:06:14    2> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:06:14    2> 	at org.elasticsearch.action.search.ClearScrollControllerTests$6.lambda$sendFreeContext$0(ClearScrollControllerTests.java:232)
17:06:14    1> 	at org.elasticsearch.action.search.ClearScrollControllerTests$6.lambda$sendFreeContext$0(ClearScrollControllerTests.java:227) ~[test/:?]
17:06:14    2> 	at java.lang.Thread.run(Thread.java:748)
17:06:14    2> REPRODUCE WITH: ./gradlew :server:test -Dtests.seed=B475D8195532593A -Dtests.class=org.elasticsearch.action.search.ClearScrollControllerTests -Dtests.method="testClearScrollIdsWithFailure" -Dtests.security.manager=true -Dtests.locale=es-CR -Dtests.timezone=America/Boise
17:06:14    1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
17:06:14    1> [2018-06-11T15:06:14,096][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearScrollIdsWithFailure]: after test
17:06:14  ERROR   0.03s J2 | ClearScrollControllerTests.testClearScrollIdsWithFailure <<< FAILURES!
17:06:14     > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=919, name=Thread-92, state=RUNNABLE, group=TGRP-ClearScrollControllerTests]
17:06:14     > 	at __randomizedtesting.SeedInfo.seed([B475D8195532593A:73C3E7A276FA0DD6]:0)
17:06:14    2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=es-CR, timezone=America/Boise
17:06:14    2> NOTE: Linux 3.16.0-4-amd64 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=16,threads=1,free=355015096,total=524812288
17:06:14     > Caused by: java.lang.AssertionError
17:06:14     > 	at __randomizedtesting.SeedInfo.seed([B475D8195532593A]:0)
17:06:14     > 	at org.elasticsearch.action.search.ClearScrollControllerTests$5.onResponse(ClearScrollControllerTests.java:195)
17:06:14     > 	at org.elasticsearch.action.search.ClearScrollControllerTests$5.onResponse(ClearScrollControllerTests.java:189)
17:06:14     > 	at org.elasticsearch.action.search.ClearScrollController.onFreedContext(ClearScrollController.java:130)
17:06:14     > 	at org.elasticsearch.action.search.ClearScrollController.lambda$cleanScrollIds$2(ClearScrollController.java:115)
17:06:14     > 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
17:06:14     > 	at org.elasticsearch.action.search.ClearScrollControllerTests$6.lambda$sendFreeContext$0(ClearScrollControllerTests.java:232)
17:06:14     > 	at java.lang.Thread.run(Thread.java:748)
17:06:14    1> [2018-06-11T15:06:14,105][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearAll]: before test
17:06:14    1> [2018-06-11T15:06:14,107][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearAll]: after test
17:06:14    1> [2018-06-11T15:06:14,109][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearScrollIds]: before test
17:06:14    1> [2018-06-11T15:06:14,127][INFO ][o.e.a.s.ClearScrollControllerTests] [testClearScrollIds]: after test
17:06:14  Completed [794/1067] on J2 in 0.06s, 3 tests, 1 error <<< FAILURES!

[jpountz]

LGTM. Great catch.

[jasontedor]

Thanks for reviewing @jpountz.