Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] IndexShardIT#testStressMaybeFlushOrRollTranslogGeneration #27770

Closed
dnhatn opened this issue Dec 12, 2017 · 1 comment
Closed

[CI] IndexShardIT#testStressMaybeFlushOrRollTranslogGeneration #27770

dnhatn opened this issue Dec 12, 2017 · 1 comment
Assignees
@dnhatn
Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI

Comments

@dnhatn
Copy link
Member

dnhatn commented Dec 12, 2017

CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.1+periodic/231/console
Log: testStressMaybeFlushOrRollTranslogGeneration.txt

FAILURE 10.1s J1 | IndexShardIT.testStressMaybeFlushOrRollTranslogGeneration <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<1> but was:<775331>
   >  at __randomizedtesting.SeedInfo.seed([9B79DBAAC8FF431B:7675EFBF2FE8B13D]:0)
   >  at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:726)
   >  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:700)
   >  at org.elasticsearch.index.shard.IndexShardIT.testStressMaybeFlushOrRollTranslogGeneration(IndexShardIT.java:464)
   >  at java.lang.Thread.run(Thread.java:748)
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.1+periodic/core/build/testrun/integTest/J1/temp/org.elasticsearch.index.shard.IndexShardIT_9B79DBAAC8FF431B-001
   >    ... 39 more
  2> Dec 11, 2017 11:41:53 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
  2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
  2> Dec 11, 2017 11:41:58 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 2 threads leaked from SUITE scope at org.elasticsearch.index.shard.IndexShardIT: 
  2>    1) Thread[id=2346, name=Thread-6, state=RUNNABLE, group=TGRP-IndexShardIT]
  2>         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
  2>         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    2) Thread[id=2345, name=Thread-5, state=RUNNABLE, group=TGRP-IndexShardIT]
  2>         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
  2>         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
  2>         at java.lang.Thread.run(Thread.java:748)
  2> Dec 11, 2017 11:41:58 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=2346, name=Thread-6, state=RUNNABLE, group=TGRP-IndexShardIT]
  2>    2) Thread[id=2345, name=Thread-5, state=RUNNABLE, group=TGRP-IndexShardIT]
  2> Dec 11, 2017 11:42:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> SEVERE: There are still zombie threads that couldn't be terminated:
  2>    1) Thread[id=2346, name=Thread-6, state=RUNNABLE, group=TGRP-IndexShardIT]
  2>         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
  2>         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    2) Thread[id=2345, name=Thread-5, state=RUNNABLE, group=TGRP-IndexShardIT]
  2>         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
  2>         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
  2>         at java.lang.Thread.run(Thread.java:748)
  2> REPRODUCE WITH: gradle :core:integTest -Dtests.seed=9B79DBAAC8FF431B -Dtests.class=org.elasticsearch.index.shard.IndexShardIT -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<0>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<470>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
  2> REPRODUCE WITH: gradle :core:integTest -Dtests.seed=9B79DBAAC8FF431B -Dtests.class=org.elasticsearch.index.shard.IndexShardIT -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1799, maxMBSortInHeap=7.258710492613572, sim=RandomSimilarity(queryNorm=false): {}, locale=en-GB, timezone=Asia/Sakhalin
  2> NOTE: Linux 4.4.0-1041-aws amd64/Oracle Corporation 1.8.0_151 (64-bit)/cpus=4,threads=3,free=213901568,total=535822336
  2> NOTE: All tests run in this JVM: [BucketSelectorIT, SimpleSearchIT, MinBucketIT, SnapshotBlocksIT, FiltersAggsRewriteIT, SimpleRoutingIT, MultiSearchIT, SimpleQueryStringIT, NaNSortingIT, IndexActionIT, SimpleIndexStateIT, MultiTermVectorsIT, BroadcastActionsIT, SuggestSearchIT, SimpleClusterStateIT, CompletionSuggestSearchIT, ExceptionRetryIT, IndexShardIT]
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<2196>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<5653>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<15351>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<38916>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<116071>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
   >  Suppressed: java.lang.AssertionError: expected:<1> but was:<247006>
   >    at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$8(IndexShardIT.java:456)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:714)
   >    ... 39 more
  1> [2017-12-12T10:41:53,835][INFO ][o.e.n.Node               ] [node_s_0] stopping ...
  1> [2017-12-12T10:41:53,858][INFO ][o.e.n.Node               ] [node_s_0] stopped
  1> [2017-12-12T10:41:53,858][INFO ][o.e.n.Node               ] [node_s_0] closing ...
  1> [2017-12-12T10:41:53,862][INFO ][o.e.n.Node               ] [node_s_0] closed
ERROR   0.00s J1 | IndexShardIT (suite) <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.elasticsearch.index.shard.IndexShardIT: 
   >    1) Thread[id=2346, name=Thread-6, state=RUNNABLE, group=TGRP-IndexShardIT]
   >         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
   >         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
   >         at java.lang.Thread.run(Thread.java:748)
   >    2) Thread[id=2345, name=Thread-5, state=RUNNABLE, group=TGRP-IndexShardIT]
   >         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
   >         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
   >         at java.lang.Thread.run(Thread.java:748)
   >  at __randomizedtesting.SeedInfo.seed([9B79DBAAC8FF431B]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   >    1) Thread[id=2346, name=Thread-6, state=RUNNABLE, group=TGRP-IndexShardIT]
   >         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
   >         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
   >         at java.lang.Thread.run(Thread.java:748)
   >    2) Thread[id=2345, name=Thread-5, state=RUNNABLE, group=TGRP-IndexShardIT]
   >         at org.elasticsearch.index.shard.IndexShardIT.lambda$testStressMaybeFlushOrRollTranslogGeneration$7(IndexShardIT.java:445)
   >         at org.elasticsearch.index.shard.IndexShardIT$$Lambda$2300/114502058.run(Unknown Source)
   >         at java.lang.Thread.run(Thread.java:748)
   >  at __randomizedtesting.SeedInfo.seed([9B79DBAAC8FF431B]:0)
@dnhatn dnhatn added :Engine >test-failure Triaged test failures from CI labels Dec 12, 2017
@dnhatn dnhatn self-assigned this Dec 12, 2017
@jasontedor
Copy link
Member

Duplicates #25773

@clintongormley clintongormley added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Engine :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clintongormley @jasontedor @dnhatn