[Backport 2.x] Improving the performance of date histogram aggregation (without any …

[jainankitk]

…sub-aggregation) (#11083)

• Adding filter based optimization logic to date histogram aggregation

Signed-off-by: Ankit Jain [email protected]

• Reading the field name for aggregation correctly

Signed-off-by: Ankit Jain [email protected]

• Adding the limit on number of buckets for filter aggregation

Signed-off-by: Ankit Jain [email protected]

• Applying the optimizations for match all query as well

Signed-off-by: Ankit Jain [email protected]

• Handling the unwrapped match all query

Signed-off-by: Ankit Jain [email protected]

• Adding logic for recursively unwrapping the query

Signed-off-by: Ankit Jain [email protected]

• Restructuring the code for making it more reusable and unit testable

Signed-off-by: Ankit Jain [email protected]

• Adding javadocs for fixing build failure

Signed-off-by: Ankit Jain [email protected]

• Fixing minor bugs in refactoring

Signed-off-by: Ankit Jain [email protected]

• Adding logic for optimizing auto date histogram

Signed-off-by: Ankit Jain [email protected]

• Fixing bugs and passing unit tests for date histogram

Signed-off-by: Ankit Jain [email protected]

• Temporarily reverting auto date histogram changes

Signed-off-by: Ankit Jain [email protected]

• Fixing spotless check bugs

Signed-off-by: Ankit Jain [email protected]

• Adding back auto date histogram and passing all unit tests

Signed-off-by: Ankit Jain [email protected]

• Fixing the integration tests for reduced collector work

Signed-off-by: Ankit Jain [email protected]

• Fixing the integration test regression

Signed-off-by: Ankit Jain [email protected]

• Addressing code review comments

Signed-off-by: Ankit Jain [email protected]

• Fixing hardbound, missing and script test cases

Signed-off-by: Ankit Jain [email protected]

• Removing collect_count validation to prevent backward compatibility tests from failing

Signed-off-by: Ankit Jain [email protected]

• Finally fixing hardbounds test case

Signed-off-by: Ankit Jain [email protected]

• Refactoring code for reusability

Signed-off-by: Ankit Jain [email protected]

---

Signed-off-by: Ankit Jain [email protected]
(cherry picked from commit 0ddbd96)

Description

Backport of #11083

Related Issues

Resolves #9310

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[jainankitk]

Depends on #11389

[github-actions]

Compatibility status:

Checks if related components are compatible with change d0d0f5d

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

✅ Gradle check result for 6fdc854: SUCCESS

[codecov]

Codecov Report

Attention: 26 lines in your changes are missing coverage. Please review.

Comparison is base (b2770de) 71.11% compared to head (d0d0f5d) 70.97%.
Report is 13 commits behind head on 2.x.

Files	Patch %	Lines
...egations/bucket/histogram/FilterRewriteHelper.java	82.41%	8 Missing and 8 partials ⚠️
.../src/main/java/org/opensearch/common/Rounding.java	66.66%	2 Missing and 2 partials ⚠️
...ions/bucket/histogram/DateHistogramAggregator.java	82.60%	3 Missing and 1 partial ⚠️
.../bucket/histogram/AutoDateHistogramAggregator.java	94.44%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #11390      +/-   ##
============================================
- Coverage     71.11%   70.97%   -0.14%     
+ Complexity    59181    59127      -54     
============================================
  Files          4869     4870       +1     
  Lines        278725   278888     +163     
  Branches      40847    40883      +36     
============================================
- Hits         198224   197951     -273     
- Misses        63816    64180     +364     
- Partials      16685    16757      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[reta]

@jainankitk could you please incorporate the changes from #11392 here so we won't be doing yet another backport, thank you (and @msfroh for the idea).

[github-actions]

❕ Gradle check result for 57b5cdf: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for d0d0f5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for d0d0f5d: SUCCESS