Add support for query profiler with concurrent aggregation

[ticheng-aws]

Description

At present, the query profiler is the total timing obtained by summing all segments. However, this does not accurately represent the detailed timing of the query, including queue time and gap time, during concurrent execution. This PR will address the problem.

A sample query response for a concurrent search with 2 segment slices:

{
    "took": 61,
    "timed_out": false,
    "num_reduce_phases": 2,
    "_shards": {...},
    "hits": {...},
    "profile":
    {
        "shards":
        [
            {
                "id": "[FFjuubYPTGSYwNLsL2ccbA][test][0]",
                "inbound_network_time_in_millis": 6,
                "outbound_network_time_in_millis": 1,
                "searches":
                [
                    {
                        "query":
                        [
                            {
                                "type": "TermQuery",
                                "description": "field1:one",
                                "time_in_nanos": 3944209,
                                "max_slice_time_in_nanos": 3944209,
                                "min_slice_time_in_nanos": 3892625,
                                "avg_slice_time_in_nanos": 3918417,
                                "breakdown":
                                {
                                    "max_match": 0,
                                    "set_min_competitive_score_count": 0,
                                    "match_count": 0,
                                    "avg_score_count": 4,
                                    "shallow_advance_count": 0,
                                    "next_doc": 50625,
                                    "min_build_scorer": 1327791,
                                    "score_count": 8,
                                    "compute_max_score_count": 0,
                                    "advance": 96583,
                                    "min_set_min_competitive_score": 0,
                                    "min_advance": 4042,
                                    "score": 96751,
                                    "avg_set_min_competitive_score_count": 0,
                                    "min_match_count": 0,
                                    "avg_score": 65438,
                                    "max_next_doc_count": 7,
                                    "max_compute_max_score_count": 0,
                                    "avg_shallow_advance": 0,
                                    "max_shallow_advance_count": 0,
                                    "set_min_competitive_score": 0,
                                    "min_build_scorer_count": 2,
                                    "next_doc_count": 8,
                                    "min_match": 0,
                                    "avg_next_doc": 26250,
                                    "compute_max_score": 0,
                                    "min_set_min_competitive_score_count": 0,
                                    "max_build_scorer": 1722750,
                                    "avg_match_count": 0,
                                    "avg_advance": 50125,
                                    "build_scorer_count": 6,
                                    "avg_build_scorer_count": 3,
                                    "min_next_doc_count": 1,
                                    "min_shallow_advance_count": 0,
                                    "max_score_count": 7,
                                    "avg_match": 0,
                                    "avg_compute_max_score": 0,
                                    "max_advance": 96208,
                                    "avg_shallow_advance_count": 0,
                                    "avg_set_min_competitive_score": 0,
                                    "avg_compute_max_score_count": 0,
                                    "avg_build_scorer": 1525270,
                                    "max_set_min_competitive_score_count": 0,
                                    "advance_count": 3,
                                    "max_build_scorer_count": 4,
                                    "shallow_advance": 0,
                                    "min_compute_max_score": 0,
                                    "max_match_count": 0,
                                    "create_weight_count": 1,
                                    "build_scorer": 1830250,
                                    "max_set_min_competitive_score": 0,
                                    "max_compute_max_score": 0,
                                    "min_shallow_advance": 0,
                                    "match": 0,
                                    "max_shallow_advance": 0,
                                    "avg_advance_count": 1,
                                    "min_next_doc": 1875,
                                    "max_advance_count": 2,
                                    "min_score": 34209,
                                    "max_next_doc": 50625,
                                    "create_weight": 472375,
                                    "avg_next_doc_count": 4,
                                    "max_score": 96667,
                                    "min_compute_max_score_count": 0,
                                    "min_score_count": 1,
                                    "min_advance_count": 1
                                }
                            }
                        ],
                        "rewrite_time": 175791,
                        "collector": [...]
                    }
                ],
                "aggregations": []
            },
            {...},
            {...}
        ]
    }
}

Related Issues

Resolves #8330 #7354

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 30m 47s

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/22416/
• CommitID: 777a7bb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git]
Compatible components: [https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 21m 42s

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/22417/
• CommitID: ea74aef
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 6644e02

Incompatible components

Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/24475/
• CommitID: 6644e02