Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reproducible Failure in RandomSamplerIT.testRandomSamplerHistogram #88108

Closed
original-brownbear opened this issue Jun 28, 2022 · 4 comments · Fixed by #88191
Closed

Reproducible Failure in RandomSamplerIT.testRandomSamplerHistogram #88108

original-brownbear opened this issue Jun 28, 2022 · 4 comments · Fixed by #88191
Assignees
@benwtrent
Labels
:Analytics/Aggregations Aggregations Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI

Comments

@original-brownbear
Copy link
Member

Fails reproducible for me on master:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.bucket.RandomSamplerIT.testRandomSamplerHistogram" -Dtests.seed=E53385B3A635A5D7 -Dtests.locale=nn-NO -Dtests.timezone=Asia/Rangoon -Druntime.java=18

ends in

  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.search.aggregations.bucket.RandomSamplerIT.testRandomSamplerHistogram" -Dtests.seed=E53385B3A635A5D7 -Dtests.locale=nn-NO -Dtests.timezone=Asia/Rangoon -Druntime.java=18
  2> java.lang.NullPointerException: Cannot invoke "java.lang.Double.doubleValue()" because the return value of "java.util.Map.get(Object)" is null
        at __randomizedtesting.SeedInfo.seed([E53385B3A635A5D7:40DCA336772BE350]:0)
        at org.elasticsearch.search.aggregations.bucket.RandomSamplerIT.testRandomSamplerHistogram(RandomSamplerIT.java:158)
  1> [2022-06-28T13:42:44,203][INFO ][o.e.s.a.b.RandomSamplerIT] [suite] [RandomSamplerIT]: cleaning up after suite
  1> [2022-06-28T13:42:44,245][INFO ][o.e.c.m.MetadataDeleteIndexService] [node_s1] [idx/-oAgidWCSni9fABrZ343RA] deleting index
  1> [2022-06-28T13:42:44,259][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,259][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,261][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,261][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,263][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,263][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,265][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,265][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,266][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,266][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,268][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,268][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,270][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,270][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,272][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,272][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,274][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,274][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,275][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,275][WARN ][o.e.d.c.s.Settings       ] [node_s0] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,278][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,278][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,280][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,281][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,283][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,283][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,285][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,285][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,287][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,287][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,289][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,289][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,291][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,291][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,292][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,293][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,294][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,294][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,295][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,295][WARN ][o.e.d.c.s.Settings       ] [node_s1] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.data_path" message="[index.data_path] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2022-06-28T13:42:44,298][INFO ][o.e.c.m.MetadataIndexTemplateService] [node_s1] removing template [random_index_template]
  1> [2022-06-28T13:42:44,305][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2022-06-28T13:42:44,307][INFO ][o.e.c.c.Coordinator      ] [node_s0] master node [{node_s1}{aCxOfFqSS9-f9aSgQLWdQA}{2xk-ap_kRRC5Udjf2tmjfA}{node_s1}{127.0.0.1}{127.0.0.1:14802}{cdfhilmrstw}] disconnected, restarting discovery
  1> [2022-06-28T13:42:44,309][INFO ][o.e.t.ClusterConnectionManager] [node_s1] transport connection to [{node_s0}{j1FKeyVmStGiR1Ms_cPlqg}{lOpr3tiZTs2jfDh32aWNgg}{node_s0}{127.0.0.1}{127.0.0.1:14801}{cdfhilmrstw}] closed by remote
  1> [2022-06-28T13:42:44,311][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2022-06-28T13:42:44,311][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2022-06-28T13:42:44,311][INFO ][o.e.c.s.MasterService    ] [node_s1] node-left[{node_s0}{j1FKeyVmStGiR1Ms_cPlqg}{lOpr3tiZTs2jfDh32aWNgg}{node_s0}{127.0.0.1}{127.0.0.1:14801}{cdfhilmrstw} reason: disconnected], term: 1, version: 24, delta: removed {{node_s0}{j1FKeyVmStGiR1Ms_cPlqg}{lOpr3tiZTs2jfDh32aWNgg}{node_s0}{127.0.0.1}{127.0.0.1:14801}{cdfhilmrstw}}
  1> [2022-06-28T13:42:44,314][INFO ][o.e.n.Node               ] [suite] closed
  1> [2022-06-28T13:42:44,315][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2022-06-28T13:42:44,316][INFO ][o.e.c.s.ClusterApplierService] [node_s1] removed {{node_s0}{j1FKeyVmStGiR1Ms_cPlqg}{lOpr3tiZTs2jfDh32aWNgg}{node_s0}{127.0.0.1}{127.0.0.1:14801}{cdfhilmrstw}}, term: 1, version: 24, reason: Publication{term=1, version=24}
  1> [2022-06-28T13:42:44,320][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2022-06-28T13:42:44,320][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2022-06-28T13:42:44,321][INFO ][o.e.n.Node               ] [suite] closed
  2> NOTE: leaving temporary files on disk at: /home/brownbear/src/elasticsearch/server/build/testrun/internalClusterTest/temp/org.elasticsearch.search.aggregations.bucket.RandomSamplerIT_E53385B3A635A5D7-002
  2> NOTE: test params are: codec=Asserting(Lucene92): {index_uuid=PostingsFormat(name=Asserting), kind=Lucene90, _id=Lucene90, type=Lucene90, kind.keyword=Lucene90}, docValues:{_seq_no=DocValuesFormat(name=Lucene90), number=DocValuesFormat(name=Asserting), monotonic=DocValuesFormat(name=Lucene90), _primary_term=DocValuesFormat(name=Asserting), kind.keyword=DocValuesFormat(name=Lucene90), _version=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=213, maxMBSortInHeap=6.6676296283317695, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=nn-NO, timezone=Asia/Rangoon
  2> NOTE: Linux 5.15.0-40-generic amd64/Private Build 18-ea (64-bit)/cpus=48,threads=1,free=367166168,total=536870912
  2> NOTE: All tests run in this JVM: [RandomSamplerIT]

Tests with failures:
 - org.elasticsearch.search.aggregations.bucket.RandomSamplerIT.testRandomSamplerHistogram
@original-brownbear original-brownbear added :Analytics/Aggregations Aggregations >test-failure Triaged test failures from CI labels Jun 28, 2022
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 28, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@benwtrent
Copy link
Member

good catch! I will take a look

@benwtrent
Copy link
Member

benwtrent commented Jun 29, 2022 •
edited
Loading

Arg, it didn't reproduce for me, but I did see a thing that could cause it to fail in some extreme circumstances. Gonna have a fix up soon.

@benwtrent
Copy link
Member

Finally able to reproduce it. Its a data set up error. Nothing fundamental with the agg or sampling. 😌

@benwtrent benwtrent mentioned this issue Jun 29, 2022
Merged
benwtrent added a commit that referenced this issue Jul 5, 2022
@benwtrent
Fix RandomSamplerIT#testRandomSamplerHistogram test input (#88191)
56befd0 
Periodically, a single 10 value is included which causes a 3rd histogram bucket to be created with very few values. Meaning sampling wouldn't get any docs that hit that bucket (especially with the randomly included deleted docs).

The failures that I saw had the 10.0 bucket with just one document.

closes: #88108
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Jul 5, 2022
@benwtrent
Fix RandomSamplerIT#testRandomSamplerHistogram test input (elastic#88191
80d4266 
)

Periodically, a single 10 value is included which causes a 3rd histogram bucket to be created with very few values. Meaning sampling wouldn't get any docs that hit that bucket (especially with the randomly included deleted docs).

The failures that I saw had the 10.0 bucket with just one document.

closes: elastic#88108
elasticsearchmachine pushed a commit that referenced this issue Jul 5, 2022
@benwtrent
Fix RandomSamplerIT#testRandomSamplerHistogram test input (#88191) (#…
4e6b148 
…88288)

Periodically, a single 10 value is included which causes a 3rd histogram bucket to be created with very few values. Meaning sampling wouldn't get any docs that hit that bucket (especially with the randomly included deleted docs).

The failures that I saw had the 10.0 bucket with just one document.

closes: #88108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benwtrent benwtrent

Labels
:Analytics/Aggregations Aggregations Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix RandomSamplerIT#testRandomSamplerHistogram test input benwtrent/elasticsearch
3 participants
@benwtrent @original-brownbear @elasticmachine