Apply fast date histogram optimization at the segment level

[bowenlan-amzn]

Description

This PR mainly aims to handle the segment level functionally match all scenario. And is a follow up of #11083 and #11505
Context: match all is one of the pre-condition before doing this filter rewrite and weight count optimization. Previously, we check this by seeing if the query is of type MatchAllQuery, FieldExists or PointRangeQuery on the same field on the shard level. All 3 scenarios mean it's safe to rewrite the date histogram configuration into set of range filter query, and later use the query weight to count how many documents are in each filter or bucket.
This PR checks if it's functionally match all at segment level if not able to create filter at shard level. This can happen if the query part of the aggregation request is not of above 3 types, while it doesn't have real effect on the current processed segment. So we can still do the filter rewrite for this segment. To make sure the results of this segment conform to the shard level bucket results. We build filter by using the shard level min max value bounds and considering the PointRangeQuery on the same field.

This PR refactors the logic that is bind to data from Aggregator into a inner class of Aggregator so those instance fields are easily accessed. While still keeps the common logic in the Helper as methods in the abstract base class or static methods.

This PR add logic to handle the existence of doc count field.

Optimization logic illustration

flowchart

    subgraph ShardLevelInit
        AggregatorInit --> C1{{"IsRewritteable (cached)"}}
        C1 -->|Yes| C7{{"CheckQueryType"}}
        C7 -->|match all, range| Build(BuildFilters):::hl
    end

	subgraph BuildFiltersDetail
        direction TB
		GetBounds --> |shard level| C5("shard bounds + PointRangeQuery")
        C5 --> GetRoundingInterval
        GetBounds --> |segment level| C6("segment bounds")
        C6 --> GetRoundingInterval
		GetRoundingInterval --> GetPreparedRounding
		GetPreparedRounding --> CreateFilters

		subgraph CreateFilters
			CountBucket --> C2{{CheckMaxNumOfBuckets}}
			C2 --> |Not Exceed| E1("CreateFilter (cached)")
		end
	end

	subgraph SegmentLevelCompute
		direction TB
		TryFilters --> C3{{CheckFilterBuildAtShardLevel}}
		C3 --> |No| C4{{CheckSegmentMatchAll}}
		C4 --> |Yes| Build2(BuildFilters):::hl
		Build2 --> BuildBuckets
		C3 --> |Yes| BuildBuckets
	end

    ShardLevelInit ~~~ SegmentLevelCompute
    SegmentLevelCompute ~~~ BuildFiltersDetail

    classDef hl fill:#f73

Loading

Related Issues

Resolves

• Apply the optimization to DocValuesFieldExistsQuery over the field (now deprecated in favor of FieldExistsQuery) #11652
• Apply the optimization if the individual segment is match all query even though the whole shard is not #11654

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change e1dbcea

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for e963981: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 85ef307: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0a78297: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for e1dbcea: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[bowenlan-amzn]

❕ Gradle check result for e1dbcea: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

#11993

[github-actions]

✅ Gradle check result for e1dbcea: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12073-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9a0a69fb530732e49834a2f2ddc3011cca82b26c
# Push it to GitHub
git push --set-upstream origin backport/backport-12073-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12073-to-2.x.

[reta]

Apparently related to this change:

[org.opensearch.search.aggregations.bucket.MinDocCountIT.testDateHistogramCountDesc {p0={"search.concurrent_segment_search.enabled":"false"}}](https://build.ci.opensearch.org/job/gradle-check/33522/testReport/junit/org.opensearch.search.aggregations.bucket/MinDocCountIT/testDateHistogramCountDesc__p0___search_concurrent_segment_search_enabled___false___/)
[org.opensearch.search.aggregations.bucket.MinDocCountIT.testDateHistogramKeyDesc {p0={"search.concurrent_segment_search.enabled":"false"}}](https://build.ci.opensearch.org/job/gradle-check/33522/testReport/junit/org.opensearch.search.aggregations.bucket/MinDocCountIT/testDateHistogramKeyDesc__p0___search_concurrent_segment_search_enabled___false___/)
[org.opensearch.search.aggregations.bucket.MinDocCountIT.testDateHistogramKeyAsc {p0={"search.concurrent_segment_search.enabled":"false"}}](https://build.ci.opensearch.org/job/gradle-check/33522/testReport/junit/org.opensearch.search.aggregations.bucket/MinDocCountIT/testDateHistogramKeyAsc__p0___search_concurrent_segment_search_enabled___false___/)
[org.opensearch.search.aggregations.bucket.MinDocCountIT.testDateHistogramCountAsc {p0={"search.concurrent_segment_search.enabled":"false"}}](https://build.ci.opensearch.org/job/gradle-check/33522/testReport/junit/org.opensearch.search.aggregations.bucket/MinDocCountIT/testDateHistogramCountAsc__p0___search_concurrent_segment_search_enabled___false___/)

Reference https://build.ci.opensearch.org/job/gradle-check/33522/
@bowenlan-amzn could you please take a look?