Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] XDocsClientYamlTestSuiteIT {yaml=en/rest-api/security/clear-cache/line_39} failed #34013

Closed
cbuescher opened this issue Sep 24, 2018 · 6 comments
Closed

[CI] XDocsClientYamlTestSuiteIT {yaml=en/rest-api/security/clear-cache/line_39} failed #34013

cbuescher opened this issue Sep 24, 2018 · 6 comments
Assignees
@nik9000
Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI v6.4.1 v8.0.0-alpha1

Comments

@cbuescher
Copy link
Member

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=amazon/20/console

At first I couldn't reproduce with:

./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=70AE4CEFD4CE334D \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_39}" \
  -Dtests.security.manager=true \
  -Dtests.locale=hi-IN \
  -Dtests.timezone=Asia/Hebron \
  -Dcompiler.java=10 \
  -Druntime.java=8

Then after a few runs I once saw the following thread dump and a failure locally, but couldn't repeat this so far:

[2018-09-24T20:18:06,313][INFO ][o.e.t.TransportService   ] [node-0] publish_address {127.0.0.1:49512}, bound_addresses {[::1]:49511}, {127.0.0.1:49512}
|    [2018-09-24T20:18:06,373][WARN ][o.e.b.BootstrapChecks    ] [node-0] HTTPS is required in order to use the token service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the token service using the [xpack.security.authc.token.enabled] setting
|
|  [jstack]
|    2018-09-24 20:18:09
|    Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0.2+13 mixed mode):
|
|    Threads class SMR info:
|    _java_thread_list=0x00007f8f32e0bfd0, length=34, elements={
|    0x00007f8f2b809000, 0x00007f8f2b02b800, 0x00007f8f2b02e800, 0x00007f8f2a840800,
|    0x00007f8f2b034000, 0x00007f8f2c0d9000, 0x00007f8f2b87f800, 0x00007f8f2d800800,
|    0x00007f8f2c0da000, 0x00007f8f2b032000, 0x00007f8f2a835000, 0x00007f8f2b2b2800,
|    0x00007f8f2f338000, 0x00007f8f2b481800, 0x00007f8f2da0c800, 0x00007f8f31059000,
|    0x00007f8f3016d800, 0x00007f8f31887000, 0x00007f8f30c0d800, 0x00007f8f31f26800,
|    0x00007f8f2b8ff000, 0x00007f8f2fb25800, 0x00007f8f2fb2a800, 0x00007f8f32077000,
|    0x00007f8f30961800, 0x00007f8f31058800, 0x00007f8f2faba000, 0x00007f8f2bc23800,
|    0x00007f8f2df08800, 0x00007f8f301a7000, 0x00007f8f2a8ac000, 0x00007f8f2f879000,
|    0x00007f8f309c2000, 0x00007f8f321e5000
|    }
|
|    "main" #1 prio=5 os_prio=31 tid=0x00007f8f2b809000 nid=0x2503 waiting on condition  [0x0000000109bf8000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a8acf5c0> (a java.util.concurrent.CountDownLatch$Sync)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:883)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly([email protected]/AbstractQueuedSynchronizer.java:1037)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly([email protected]/AbstractQueuedSynchronizer.java:1343)
|       at java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:232)
|       at org.elasticsearch.node.Node.start(Node.java:727)
|       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:264)
|       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:335)
|       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
|       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
|       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
|       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
|       at org.elasticsearch.cli.Command.main(Command.java:90)
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
|       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
|
|    "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f8f2b02b800 nid=0x3903 waiting on condition  [0x0000700003ff0000]
|       java.lang.Thread.State: RUNNABLE
|       at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method)
|       at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:174)
|       at java.lang.ref.Reference.access$000([email protected]/Reference.java:44)
|       at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:138)
|
|    "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f8f2b02e800 nid=0x3c03 in Object.wait()  [0x00007000040f3000]
|       java.lang.Thread.State: WAITING (on object monitor)
|       at java.lang.Object.wait([email protected]/Native Method)
|       - waiting on <0x00000007aaac3810> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:151)
|       - waiting to re-lock in wait() <0x00000007aaac3810> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:172)
|       at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Finalizer.java:216)
|
|    "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f8f2a840800 nid=0xa803 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f8f2b034000 nid=0x5a03 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       Compiling: 5203 %     4       org.apache.lucene.util.automaton.MinimizationOperations::minimize @ 777 (1439 bytes)
|
|    "C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f8f2c0d9000 nid=0xa703 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f8f2b87f800 nid=0x5d03 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f8f2d800800 nid=0xa303 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|       No compile task
|
|    "Sweeper thread" #9 daemon prio=9 os_prio=31 tid=0x00007f8f2c0da000 nid=0xa003 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "Service Thread" #10 daemon prio=9 os_prio=31 tid=0x00007f8f2b032000 nid=0x9f03 runnable  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "Common-Cleaner" #11 daemon prio=8 os_prio=31 tid=0x00007f8f2a835000 nid=0x6103 in Object.wait()  [0x000070000490b000]
|       java.lang.Thread.State: TIMED_WAITING (on object monitor)
|       at java.lang.Object.wait([email protected]/Native Method)
|       - waiting on <0x00000007aaaca998> (a java.lang.ref.ReferenceQueue$Lock)
|       at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:151)
|       - waiting to re-lock in wait() <0x00000007aaaca998> (a java.lang.ref.ReferenceQueue$Lock)
|       at jdk.internal.ref.CleanerImpl.run([email protected]/CleanerImpl.java:148)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|       at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:134)
|
|    "process reaper" #16 daemon prio=10 os_prio=31 tid=0x00007f8f2b2b2800 nid=0x6303 runnable  [0x0000700004a35000]
|       java.lang.Thread.State: RUNNABLE
|       at java.lang.ProcessHandleImpl.waitForProcessExit0([email protected]/Native Method)
|       at java.lang.ProcessHandleImpl.access$000([email protected]/ProcessHandleImpl.java:50)
|       at java.lang.ProcessHandleImpl$1.run([email protected]/ProcessHandleImpl.java:138)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1135)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[timer]]" #19 daemon prio=5 os_prio=31 tid=0x00007f8f2f338000 nid=0x1007 waiting on condition  [0x0000700003243000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep([email protected]/Native Method)
|       at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:549)
|
|    "elasticsearch[node-0][scheduler][T#1]" #20 daemon prio=5 os_prio=31 tid=0x00007f8f2b481800 nid=0x6503 waiting on condition  [0x0000700004b38000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007abc067b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|       at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:2117)
|       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
|       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "Connection evictor" #21 daemon prio=5 os_prio=31 tid=0x00007f8f2da0c800 nid=0x9803 waiting on condition  [0x0000700004c3b000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep([email protected]/Native Method)
|       at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "ticker-schedule-trigger-engine" #22 daemon prio=5 os_prio=31 tid=0x00007f8f31059000 nid=0x9603 waiting on condition  [0x0000700004d3e000]
|       java.lang.Thread.State: TIMED_WAITING (sleeping)
|       at java.lang.Thread.sleep([email protected]/Native Method)
|       at org.elasticsearch.xpack.watcher.trigger.schedule.engine.TickerScheduleTriggerEngine$Ticker.run(TickerScheduleTriggerEngine.java:171)
|
|    "ml-cpp-log-tail-thread" #23 daemon prio=5 os_prio=31 tid=0x00007f8f3016d800 nid=0x6a03 runnable  [0x0000700004e41000]
|       java.lang.Thread.State: RUNNABLE
|       at java.io.FileInputStream.readBytes([email protected]/Native Method)
|       at java.io.FileInputStream.read([email protected]/FileInputStream.java:258)
|       at org.elasticsearch.xpack.ml.job.process.logging.CppLogMessageHandler.tailStream(CppLogMessageHandler.java:105)
|       at org.elasticsearch.xpack.ml.job.process.NativeController.lambda$tailLogsInThread$0(NativeController.java:69)
|       at org.elasticsearch.xpack.ml.job.process.NativeController$$Lambda$1831/1890190272.run(Unknown Source)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#1]" #24 daemon prio=5 os_prio=31 tid=0x00007f8f31887000 nid=0x6d03 waiting on condition  [0x0000700004f44000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take([email protected]/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#2]" #25 daemon prio=5 os_prio=31 tid=0x00007f8f30c0d800 nid=0x6f03 waiting on condition  [0x0000700005047000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a8a3bbc8> (a java.util.concurrent.CompletableFuture$Signaller)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1796)
|       at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3156)
|       at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1823)
|       at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:1998)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.pingAndWait(ZenDiscovery.java:1044)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.findMaster(ZenDiscovery.java:887)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:441)
|       at org.elasticsearch.discovery.zen.ZenDiscovery.access$2500(ZenDiscovery.java:89)
|       at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1246)
|       at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1135)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#3]" #26 daemon prio=5 os_prio=31 tid=0x00007f8f31f26800 nid=0x7003 waiting on condition  [0x000070000514a000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take([email protected]/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][generic][T#4]" #27 daemon prio=5 os_prio=31 tid=0x00007f8f2b8ff000 nid=0x9003 waiting on condition  [0x000070000524d000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007abd01438> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:743)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.take([email protected]/LinkedTransferQueue.java:1367)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[transport_server_worker.default]][T#1]" #28 daemon prio=5 os_prio=31 tid=0x00007f8f2fb25800 nid=0x8e03 runnable  [0x0000700005350000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a8c8b0f0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a8c8b100> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a8c8b0a0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[transport_server_worker.default]][T#2]" #29 daemon prio=5 os_prio=31 tid=0x00007f8f2fb2a800 nid=0x7403 runnable  [0x0000700005453000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a8914a60> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a8914a70> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a8914a10> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#1]" #30 daemon prio=5 os_prio=31 tid=0x00007f8f32077000 nid=0x8c03 waiting on condition  [0x0000700005556000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll([email protected]/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#2]" #31 daemon prio=5 os_prio=31 tid=0x00007f8f30961800 nid=0x7903 waiting on condition  [0x0000700005659000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll([email protected]/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][clusterApplierService#updateTask][T#1]" #32 daemon prio=5 os_prio=31 tid=0x00007f8f31058800 nid=0x7b03 waiting on condition  [0x000070000575c000]
|       java.lang.Thread.State: WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a8bf7400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|       at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
|       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2075)
|       at java.util.concurrent.PriorityBlockingQueue.take([email protected]/PriorityBlockingQueue.java:550)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1061)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][[unicast_connect]][T#3]" #33 daemon prio=5 os_prio=31 tid=0x00007f8f2faba000 nid=0x7d03 waiting on condition  [0x000070000585f000]
|       java.lang.Thread.State: TIMED_WAITING (parking)
|       at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
|       - parking to wait for  <0x00000007a9397eb8> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
|       at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
|       at java.util.concurrent.LinkedTransferQueue.awaitMatch([email protected]/LinkedTransferQueue.java:740)
|       at java.util.concurrent.LinkedTransferQueue.xfer([email protected]/LinkedTransferQueue.java:684)
|       at java.util.concurrent.LinkedTransferQueue.poll([email protected]/LinkedTransferQueue.java:1375)
|       at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1060)
|       at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1121)
|       at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#2]" #35 daemon prio=5 os_prio=31 tid=0x00007f8f2bc23800 nid=0x8903 runnable  [0x0000700005962000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a9414280> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a9414270> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a94141d0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#1]" #34 daemon prio=5 os_prio=31 tid=0x00007f8f2df08800 nid=0x8103 runnable  [0x0000700005a65000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a919f1a0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a919f190> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a919f0f0> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#4]" #36 daemon prio=5 os_prio=31 tid=0x00007f8f301a7000 nid=0x8603 runnable  [0x0000700005b68000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a91a4130> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a4120> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a4080> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#3]" #37 daemon prio=5 os_prio=31 tid=0x00007f8f2a8ac000 nid=0x8403 runnable  [0x0000700005c6b000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a91a19b0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a19a0> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a1900> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#5]" #38 daemon prio=5 os_prio=31 tid=0x00007f8f2f879000 nid=0xac03 runnable  [0x0000700005d6e000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a91a68b0> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a68a0> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a6800> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "elasticsearch[node-0][transport_client_boss][T#6]" #39 daemon prio=5 os_prio=31 tid=0x00007f8f309c2000 nid=0x15303 runnable  [0x0000700005e71000]
|       java.lang.Thread.State: RUNNABLE
|       at sun.nio.ch.KQueueArrayWrapper.kevent0([email protected]/Native Method)
|       at sun.nio.ch.KQueueArrayWrapper.poll([email protected]/KQueueArrayWrapper.java:196)
|       at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:116)
|       at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:89)
|       - locked <0x00000007a91a9030> (a sun.nio.ch.Util$2)
|       - locked <0x00000007a91a9020> (a java.util.Collections$UnmodifiableSet)
|       - locked <0x00000007a91a8f80> (a sun.nio.ch.KQueueSelectorImpl)
|       at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:100)
|       at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
|       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
|       at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
|       at java.lang.Thread.run([email protected]/Thread.java:844)
|
|    "Attach Listener" #40 daemon prio=9 os_prio=31 tid=0x00007f8f321e5000 nid=0x15103 waiting on condition  [0x0000000000000000]
|       java.lang.Thread.State: RUNNABLE
|
|    "VM Thread" os_prio=31 tid=0x00007f8f2b039000 nid=0x3703 runnable
|
|    "GC Thread#0" os_prio=31 tid=0x00007f8f2b000000 nid=0x5103 runnable
|
|    "GC Thread#1" os_prio=31 tid=0x00007f8f2c00c000 nid=0x2d03 runnable
|
|    "GC Thread#2" os_prio=31 tid=0x00007f8f2c00d000 nid=0x4f03 runnable
|
|    "GC Thread#3" os_prio=31 tid=0x00007f8f2c00d800 nid=0x4d03 runnable
|
|    "GC Thread#4" os_prio=31 tid=0x00007f8f2c00e800 nid=0x2f03 runnable
|
|    "GC Thread#5" os_prio=31 tid=0x00007f8f2c00f000 nid=0x3103 runnable
|
|    "GC Thread#6" os_prio=31 tid=0x00007f8f2c010000 nid=0x4c03 runnable
|
|    "GC Thread#7" os_prio=31 tid=0x00007f8f2b812000 nid=0x3403 runnable
|
|    "CMS Main Thread" os_prio=31 tid=0x00007f8f2b81d800 nid=0x4803 runnable
|
|    "CMS Thread#0" os_prio=31 tid=0x00007f8f2c010800 nid=0x4b03 runnable
|
|    "CMS Thread#1" os_prio=31 tid=0x00007f8f2a802800 nid=0x3603 runnable
|
|    "VM Periodic Task Thread" os_prio=31 tid=0x00007f8f2a801000 nid=0x9b03 waiting on condition
|
|    JNI global references: 74
|
|-----------------------------------------

> Task :x-pack:docs:integTestCluster#wait FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':x-pack:docs:integTestCluster#wait'.
> Failed to start elasticsearch: timed out after 30 seconds

I'm currently not able to log into the machine (ssh timeouts) to look if there are similar things in the cluster logs, the failure I see in the test log is:

18:01:10   1> [2018-09-24T19:00:39,963][ERROR][o.e.s.XDocsClientYamlTestSuiteIT] [test] This failing test was generated by documentation starting at en/rest-api/security/clear-cache.asciidoc:line_39. It may include many snippets. See Elasticsearch's docs/README.asciidoc for an explanation of test generation.
18:01:10 ERROR   0.45s | XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/clear-cache/line_39} <<< FAILURES!
18:01:10    > Throwable #1: org.elasticsearch.test.rest.yaml.ClientYamlTestResponseException: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:46301], URI [/_xpack/security/user?error_trace=true], status line [HTTP/1.1 503 Service Unavailable]
18:01:10    > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"stack_trace":"Failed to execute phase [query], all shards failed\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)\n\tat org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:106)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:277)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:442)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1121)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1218)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1202)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)\n\tat org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:103)\n\tat org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:346)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:340)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:334)\n\tat org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1073)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:844)\n"},"status":503}
@cbuescher cbuescher added >test-failure Triaged test failures from CI v7.0.0 :Security/Security Security issues without another label labels Sep 24, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@droberts195
Copy link
Contributor

Another test in the same docs file failed in the same way in 6.4 and I suspect the root cause is the same.

The repro command is this, although it didn't reproduce locally for me:

./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=6668AB67F4819461 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_47}" \
  -Dtests.security.manager=true \
  -Dtests.locale=th \
  -Dtests.timezone=Pacific/Galapagos

The error in the test log is this:

   > Throwable #1: org.elasticsearch.test.rest.yaml.ClientYamlTestResponseException: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:40403], URI [/_xpack/security/user?error_trace=true], status line [HTTP/1.1 503 Service Unavailable]
   > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[],"stack_trace":"Failed to execute phase [query], all shards failed\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)\n\tat org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222)\n\tat org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176)\n\tat org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48)\n\tat org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:526)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1068)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1165)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)\n\tat org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:384)\n\tat org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)\n\tat org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329)\n\tat org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:844)\n"},"status":503}

These are the log messages in the server log around the time the error occurred:

[2018-09-26T06:05:37,760][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.security-6/-VvwEk-qQyCVyi-EtdpUTg] deleting index
[2018-09-26T06:05:37,917][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.security-6] creating index, cause [api], templates [security-index-template], shards [1]/[0], mappings [doc]
[2018-09-26T06:05:37,946][DEBUG][o.e.a.s.TransportSearchAction] [node-0] All shards failed for phase: [query]
[2018-09-26T06:05:37,974][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-6][0]] ...]).
[2018-09-26T06:05:38,094][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-0] [.security-6/WAbCr-IhRKqL_sRxyHbn6A] deleting index
[2018-09-26T06:05:38,385][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-0] [.security-6] creating index, cause [api], templates [security-index-template], shards [1]/[0], mappings [doc]
[2018-09-26T06:05:38,486][INFO ][o.e.c.r.a.AllocationService] [node-0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-6][0]] ...]).

The timestamp on the error itself is 2018-09-26T06:05:38,234, so during the period between when the .security-6 was deleted and when it was recreated. So I suspect the problem here is that the docs tests are repeatedly deleting and recreating the .security-6 index without making sure that the test code runs while the index exists.

@droberts195 droberts195 mentioned this issue Sep 26, 2018
Closed
@nik9000
Copy link
Member

nik9000 commented Oct 4, 2018

We believe this is caused because the tests clean out the security index after each test but don't wait for it to be recreated before the test. I've started working on cleaning up some of the other x-pack test cleanup code and eventually I'll get to security. And when I do, this issue is waiting for me.

@DaveCTurner DaveCTurner mentioned this issue Oct 15, 2018
Closed
@andrershov
Copy link
Contributor

Reproduced again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/171/console

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=CA88C7EBB4D1EE93 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/rest-api/security/clear-cache/line_39}" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-QA \
  -Dtests.timezone=Greenwich \
  -Dcompiler.java=11 \
  -Druntime.java=8

@jasontedor jasontedor added v8.0.0 and removed v7.0.0 labels Feb 6, 2019
@andyb-elastic
Copy link
Contributor

I ran the tests for this doc page in a loop a couple hundred times and couldn't reproduce. Here's the failures for security docs pages with stack traces that look like the ones here. It seems like there were a handful of them in fall 2018 and none since then

@jkakavas
Copy link
Member

As these haven't failed since late October last year, it looks related to #33864. Could it be that the failures here were also caused/affected by the watcher issue that was resolved in #35271 ?

I'm going to close this for now, we can reopen if any such failure re-emerges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nik9000 nik9000

Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI v6.4.1 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@nik9000 @andrershov @jakelandis @jasontedor @droberts195 @jkakavas @cbuescher @elasticmachine @andyb-elastic