Composite fails to run on sorted indices #72078

benwtrent · 2021-04-22T11:27:22Z

Elasticsearch version (bin/elasticsearch --version):
master, 7.13
Description of the problem including expected versus actual behavior:
Composite aggs should work with sorted indices
Steps to reproduce:

Have an index sorted in ascending order by a timestamp
have a fairly large cardinality term group_by
call composite aggs with an after indicated
watch blow up with:

"failed_shards" : [
      {
        "shard" : 0,
        "index" : "filebeat-nginx-elasticco-anon-2017-sorted",
        "node" : "3hlORREsS-WP8illLYkF0g",
        "reason" : {
          "type" : "illegal_state_exception",
          "reason" : "getLeafCollector should return a noop collector instead of throw CollectionTerminatedException",
          "caused_by" : {
            "type" : "collection_terminated_exception",
            "reason" : null
          }
        }
      }
    ]

I am thinking the leaf collector noop check is too broad. It appears to be catching and rethrowing even if the collect method in the collector throws.

elasticsearch/server/src/main/java/org/elasticsearch/search/aggregations/bucket/composite/CompositeValuesCollectorQueue.java

Line 278 in 5f471f4

throw new CollectionTerminatedException();

Provide logs (if relevant):

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:663) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:696) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:624) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1168) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1277) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1251) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.ElasticsearchException$1: getLeafCollector should return a noop collector instead of throw CollectionTerminatedException
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:633) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:382) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	... 21 more
Caused by: java.lang.IllegalStateException: getLeafCollector should return a noop collector instead of throw CollectionTerminatedException
	at org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:101) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:204) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:189) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:336) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1410) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1473) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:424) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1479) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:421) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	... 6 more
Caused by: org.apache.lucene.search.CollectionTerminatedException
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesCollectorQueue.addIfCompetitive(CompositeValuesCollectorQueue.java:291) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator$3.collect(CompositeAggregator.java:435) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.bucket.composite.GlobalOrdinalValuesSource$1.collect(GlobalOrdinalValuesSource.java:131) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.bucket.composite.LongValuesSource$1.collect(LongValuesSource.java:165) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:116) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.processLeafFromQuery(CompositeAggregator.java:370) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.getLeafCollector(CompositeAggregator.java:409) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:207) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:94) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.aggregations.MultiBucketCollector$1.getLeafCollector(MultiBucketCollector.java:75) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:124) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:204) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:189) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27]
	at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:336) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$25(IndicesService.java:1410) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$26(IndicesService.java:1473) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:164) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:424) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:110) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1479) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:421) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	... 6 more

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-04-22T11:27:24Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

nik9000 · 2021-04-22T11:30:09Z

This looks like one of my bugs. I'll have a look at it today. That does mean we don't have any tests for this case. That'll be fun to track down.

…

On Thu, Apr 22, 2021, 07:27 Elastic Machine ***@***.***> wrote: Pinging @elastic/es-analytics-geo <https://github.com/orgs/elastic/teams/es-analytics-geo> (Team:Analytics) — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#72078 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUXIVBRIADV2DUSNR7OTTTKAB3FANCNFSM43MIJGYA> .

I broke composite early termination when reworking how aggregations' contact for `getLeafCollector` around early termination in elastic#70320. We didn't see it in our tests because we weren't properly emulating the aggregation collection stage. This fixes early termination by adhering to the new contract and adds more tests. Closes elastic#72078 Co-authored-by: Benjamin Trent <[email protected]>

I broke composite early termination when reworking how aggregations' contact for `getLeafCollector` around early termination in #70320. We didn't see it in our tests because we weren't properly emulating the aggregation collection stage. This fixes early termination by adhering to the new contract and adds more tests. Closes #72078 Co-authored-by: Benjamin Trent <[email protected]>

I broke composite early termination when reworking how aggregations' contact for `getLeafCollector` around early termination in elastic#70320. We didn't see it in our tests because we weren't properly emulating the aggregation collection stage. This fixes early termination by adhering to the new contract and adds more tests. Closes elastic#72078 Co-authored-by: Benjamin Trent <[email protected]>

I broke composite early termination when reworking how aggregations' contact for `getLeafCollector` around early termination in #70320. We didn't see it in our tests because we weren't properly emulating the aggregation collection stage. This fixes early termination by adhering to the new contract and adds more tests. Closes #72078 Co-authored-by: Benjamin Trent <[email protected]>

benwtrent added >bug :Analytics/Aggregations Aggregations labels Apr 22, 2021

elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Apr 22, 2021

benwtrent mentioned this issue Apr 22, 2021

Fixes bug when composite aggs when the index is sorted #72086

Closed

nik9000 self-assigned this Apr 22, 2021

nik9000 mentioned this issue Apr 22, 2021

Fix composite early termination on sorted #72101

Merged

nik9000 closed this as completed in #72101 Apr 22, 2021

nik9000 mentioned this issue Apr 22, 2021

Fix composite early termination on sorted (backport of #72101) #72127

Merged

nik9000 mentioned this issue Apr 22, 2021

Fix composite early termination on sorted (backport of #72101) #72128

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Composite fails to run on sorted indices #72078

Composite fails to run on sorted indices #72078

benwtrent commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

nik9000 commented Apr 22, 2021 via email

Composite fails to run on sorted indices #72078

Composite fails to run on sorted indices #72078

Comments

benwtrent commented Apr 22, 2021

elasticmachine commented Apr 22, 2021

nik9000 commented Apr 22, 2021 via email