Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] QueryApiKeyIT testPagination failing #76542

Closed
ywangd opened this issue Aug 16, 2021 · 3 comments · Fixed by #76654
Closed

[CI] QueryApiKeyIT testPagination failing #76542

ywangd opened this issue Aug 16, 2021 · 3 comments · Fixed by #76654
Assignees
@ywangd
Labels
:Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@ywangd
Copy link
Member

ywangd commented Aug 16, 2021

Not reproducible locally after 20+ attempts

Build scan:
https://gradle-enterprise.elastic.co/s/aiu2ymtwfogys/tests/:x-pack:plugin:security:qa:security-basic:javaRestTest/org.elasticsearch.xpack.security.QueryApiKeyIT/testPagination

Reproduction line:
./gradlew ':x-pack:plugin:security:qa:security-basic:javaRestTest' --tests "org.elasticsearch.xpack.security.QueryApiKeyIT.testPagination" -Dtests.seed=D6C161C603B9D7A7 -Dtests.locale=fr-BE -Dtests.timezone=America/Chihuahua -Druntime.java=8

Applicable branches:
7.x

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.QueryApiKeyIT&tests.test=testPagination

Failure excerpt:

java.lang.AssertionError: 
Expected: "k-07"
     but: was "k-08"

  at __randomizedtesting.SeedInfo.seed([D6C161C603B9D7A7:BEE1A8EB7F7B7ECE]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.xpack.security.QueryApiKeyIT.testPagination(QueryApiKeyIT.java:248)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)
@ywangd ywangd added :Security/Security Security issues without another label >test-failure Triaged test failures from CI labels Aug 16, 2021
@elasticmachine elasticmachine added the Team:Security Meta label for security team label Aug 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@ywangd ywangd self-assigned this Aug 16, 2021
ywangd added a commit to ywangd/elasticsearch that referenced this issue Aug 16, 2021
@ywangd
[Test] Improve test assert for better message on failure
abc0752 
Relates: elastic#76542
@ywangd ywangd mentioned this issue Aug 16, 2021
Merged
ywangd added a commit that referenced this issue Aug 17, 2021
@ywangd
[Test] Improve assertions for better messages on failure (#76543)
add0b77 
This PR adjusts assertions in testPagination so that more detailed messages
would be made available on failures. For example, if the sorted keys do not
match the expected order, it now shows all keys for better context instead of
just the single mismatched key.

Relates: #76542
ywangd added a commit to ywangd/elasticsearch that referenced this issue Aug 17, 2021
@ywangd
[Test] Improve assertions for better messages on failure (elastic#76543)
3ea2fc1 
This PR adjusts assertions in testPagination so that more detailed messages
would be made available on failures. For example, if the sorted keys do not
match the expected order, it now shows all keys for better context instead of
just the single mismatched key.

Relates: elastic#76542
@ywangd ywangd mentioned this issue Aug 17, 2021
Merged
ywangd added a commit that referenced this issue Aug 17, 2021
@ywangd
[Test] Improve assertions for better messages on failure (#76543) (#7…
1b7542d 
…6596)

* [Test] Improve assertions for better messages on failure (#76543)

This PR adjusts assertions in testPagination so that more detailed messages
would be made available on failures. For example, if the sorted keys do not
match the expected order, it now shows all keys for better context instead of
just the single mismatched key.

Relates: #76542

* fix for 7.x quirks
@dimitris-athanasiou
Copy link
Contributor

Another failure in a PR of mine: https://gradle-enterprise.elastic.co/s/46sa5efqr3xcu/tests/:x-pack:plugin:security:qa:security-basic:javaRestTest/org.elasticsearch.xpack.security.QueryApiKeyIT/testPagination?top-execution=1

ywangd added a commit to ywangd/elasticsearch that referenced this issue Aug 18, 2021
@ywangd
[Test] More robust assertions for sorting and pagination
556bb08 
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: elastic#76542
@ywangd ywangd mentioned this issue Aug 18, 2021
Merged
@DaveCTurner
Copy link
Contributor

https://gradle-enterprise.elastic.co/s/bt46tjruvjtay/console-log?task=:x-pack:plugin:security:qa:security-basic:javaRestTest looks like another

ywangd added a commit that referenced this issue Aug 24, 2021
@ywangd
[Test] More robust assertions for sorting and pagination (#76654)
a5a1c6a 
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: #76542
ywangd added a commit to ywangd/elasticsearch that referenced this issue Aug 24, 2021
@ywangd
[Test] More robust assertions for sorting and pagination (elastic#76654)
aeec15f 
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: elastic#76542
@ywangd ywangd mentioned this issue Aug 24, 2021
Merged
ywangd added a commit to ywangd/elasticsearch that referenced this issue Aug 24, 2021
@ywangd
[Test] More robust assertions for sorting and pagination (elastic#76654)
5ee8a77 
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: elastic#76542
@ywangd ywangd mentioned this issue Aug 24, 2021
Merged
elasticsearchmachine pushed a commit that referenced this issue Aug 24, 2021
@ywangd
[Test] More robust assertions for sorting and pagination (#76654) (#7…
d74d559 
…6859)

Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: #76542
elasticsearchmachine pushed a commit that referenced this issue Aug 24, 2021
@ywangd
[Test] More robust assertions for sorting and pagination (#76654) (#7…
14f22ef 
…6858)

Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: #76542
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywangd ywangd

Labels
:Security/Security Security issues without another label Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Test] More robust assertions for sorting and pagination ywangd/elasticsearch
4 participants
@ywangd @DaveCTurner @dimitris-athanasiou @elasticmachine