Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MvConcatTests testCrankyEvaluateBlockWithoutNulls {TestCase=KEYWORD/8 KEYWORD} failing #101969

Closed
ldematte opened this issue Nov 9, 2023 · 5 comments · Fixed by #102350
Closed

[CI] MvConcatTests testCrankyEvaluateBlockWithoutNulls {TestCase=KEYWORD/8 KEYWORD} failing #101969

ldematte opened this issue Nov 9, 2023 · 5 comments · Fixed by #102350
Assignees
@nik9000
Labels
:Analytics/ES|QL AKA ESQL blocker Team:QL (Deprecated) Meta label for query languages team >test-failure Triaged test failures from CI
Milestone
8.12

Comments

@ldematte
Copy link
Contributor

ldematte commented Nov 9, 2023

Test failing with different args in the same run, all related to index out of bounds

Build scan:
https://gradle-enterprise.elastic.co/s/3bik4qiof2dei/tests/:x-pack:plugin:esql:test/org.elasticsearch.xpack.esql.expression.function.scalar.multivalue.MvConcatTests/testCrankyEvaluateBlockWithoutNulls%20%7BTestCase=KEYWORD%2F8%20KEYWORD%7D
Reproduction line:

./gradlew ':x-pack:plugin:esql:test' --tests "org.elasticsearch.xpack.esql.expression.function.scalar.multivalue.MvConcatTests.testCrankyEvaluateBlockWithoutNulls {TestCase=KEYWORD/8 KEYWORD}" -Dtests.seed=F0089FCA0A9B6D21 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Africa/Mbabane -Druntime.java=21

Applicable branches:
8.11

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.esql.expression.function.scalar.multivalue.MvConcatTests&tests.test=testCrankyEvaluateBlockWithoutNulls%20%7BTestCase%3DKEYWORD/8%20KEYWORD%7D

Failure excerpt:

java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2

  at __randomizedtesting.SeedInfo.seed([F0089FCA0A9B6D21:9AE69939A4A7D1CF]:0)
  at org.elasticsearch.common.util.BigByteArray.get(BigByteArray.java:70)
  at org.elasticsearch.common.util.MockBigArrays$ByteArrayWrapper.get(MockBigArrays.java:388)
  at org.elasticsearch.common.util.BytesRefArray.get(BytesRefArray.java:109)
  at org.elasticsearch.compute.data.BytesRefArrayBlock.getBytesRef(BytesRefArrayBlock.java:50)
  at org.elasticsearch.xpack.esql.expression.function.scalar.multivalue.MvConcat$MvConcatEvaluator.eval(MvConcat.java:136)
  at org.elasticsearch.xpack.esql.expression.function.AbstractFunctionTestCase.testEvaluateBlock(AbstractFunctionTestCase.java:370)
  at org.elasticsearch.xpack.esql.expression.function.AbstractFunctionTestCase.testCrankyEvaluateBlockWithoutNulls(AbstractFunctionTestCase.java:283)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
@ldematte ldematte added :Analytics/ES|QL AKA ESQL >test-failure Triaged test failures from CI labels Nov 9, 2023
@elasticsearchmachine elasticsearchmachine added blocker Team:QL (Deprecated) Meta label for query languages team labels Nov 9, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

@costin costin added this to the 8.12 milestone Nov 16, 2023
@nik9000 nik9000 self-assigned this Nov 16, 2023
@nik9000
Copy link
Member

nik9000 commented Nov 16, 2023

This is caused by stuffing empty strings somehow. I think it has to do with putting them at the boundary of the bytes page that backs BigByteArray.

@nik9000
Copy link
Member

nik9000 commented Nov 16, 2023

This seems pretty rare. The test data is a multivalued string field that's exactly 64 byte long:

 1 [61],
 5 [f0 9f 8a 8c 61], 
22 [e2 8e 84 e2 8e bd e2 8c 83 e2 8f 9e e2 8f b2 e2 8e 86 e2 8e a7 61], 
 0 [], 
 1 [61], 
13 [f0 ab 9b 84 f0 aa b2 86 f0 ab 95 a5 61],
22 [e1 aa 97 e1 a8 bc e1 aa 93 e1 aa a6 e1 a8 b1 e1 a9 ac e1 a9 b7 61], 
 0 []

I think you have to have a value that's a power of two and ends in an empty value. Checking more.

@nik9000
Copy link
Member

nik9000 commented Nov 16, 2023

And I'm able to reproduce it directly against BytesRefArray with this data:

  0 [],
  9 [4f 73 76 50 4a 53 6b 68 41],
  0 [],
  9 [75 79 57 62 6f 46 6f 56 43],
  5 [53 64 52 65 64],
  9 [4a 43 4f 72 50 6b 59 62 55],
  0 [],
  0 [],
  0 [],
  0 []

Which is 32 bytes long and ends with an empty value.

@nik9000 nik9000 mentioned this issue Nov 18, 2023
Merged
sabi0 pushed a commit to sabi0/elasticsearch that referenced this issue Nov 20, 2023
@nik9000
ESQL: Fix rare bug with empty string (elastic#102350)
e9dee63 
This fixes a rare bug that occurs when an empty value lands on the page
boundary of the last page. We only allocate the last page if we need
some bytes from it. So if you add an empty string to the as the last
entry in a BytesRefBlock we don't allocate a whole new slab of bytes to
hold the empty string. But, without this change, we attempt to read from
the unallocated array. We try to read 0 bytes from it, but still. That's
a read past the end of the array.

Closes elastic#101969
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nik9000 nik9000

Labels
:Analytics/ES|QL AKA ESQL blocker Team:QL (Deprecated) Meta label for query languages team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
8.12
Development

Successfully merging a pull request may close this issue.

ESQL: Fix rare bug with empty string nik9000/elasticsearch
4 participants
@costin @nik9000 @ldematte @elasticsearchmachine