[Backport 2.x] [Search Pipelines] Add stats for search pipelines

[mingshl]

Description

manual back-porting #8053 and resolve conflict

Related Issues

Resolves #6723

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18793/
• CommitID: a74d58d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18825/
• CommitID: a74d58d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18849/
• CommitID: 209a00a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[mingshl]

Need help in rerunning gradle check

[msfroh]

It looks like the Gradle check is failing because of the 1.3.11 release -- the buildBwcLinuxTar target is now building 1.3.12?

[msfroh]

Note that to avoid breaking backward compatibility tests on main, we will need to update the version checks on main to also check for versions on or after 2.9.0. Please create the PR for that and link it from this PR. They basically need to go in together (until we come up with a better solution).

See #8070 for discussion.

[mingshl]

Note that to avoid breaking backward compatibility tests on main, we will need to update the version checks on main to also check for versions on or after 2.9.0. Please create the PR for that and link it from this PR. They basically need to go in together (until we come up with a better solution).

See #8070 for discussion.

created #8416 to update the version checks on main

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload

• URL: https://build.ci.opensearch.org/job/gradle-check/19156/
• CommitID: 0d648c8
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8376 (7a40bdc) into 2.x (9b9718f) will increase coverage by 0.04%.
The diff coverage is 87.93%.

@@             Coverage Diff              @@
##                2.x    #8376      +/-   ##
============================================
+ Coverage     70.47%   70.52%   +0.04%     
- Complexity    56953    57039      +86     
============================================
  Files          4737     4740       +3     
  Lines        270103   270416     +313     
  Branches      39905    39940      +35     
============================================
+ Hits         190345   190699     +354     
+ Misses        63404    63348      -56     
- Partials      16354    16369      +15     

Impacted Files	Coverage Δ
...min/cluster/stats/TransportClusterStatsAction.java	70.83% <ø> (ø)
...src/main/java/org/opensearch/node/NodeService.java	73.49% <0.00%> (-0.90%)	⬇️
...rch/action/admin/cluster/node/stats/NodeStats.java	47.66% <33.33%> (-0.71%)	⬇️
.../action/admin/cluster/stats/ClusterStatsNodes.java	55.00% <55.55%> (ø)
.../org/opensearch/common/metrics/OperationStats.java	78.78% <78.78%> (ø)
.../java/org/opensearch/search/pipeline/Pipeline.java	87.83% <85.96%> (-1.82%)	⬇️
...pensearch/search/pipeline/SearchPipelineStats.java	89.44% <89.44%> (ø)
...pensearch/search/pipeline/PipelineWithMetrics.java	90.90% <90.90%> (ø)
...main/java/org/opensearch/ingest/IngestService.java	83.37% <93.75%> (ø)
...on/admin/cluster/node/stats/NodesStatsRequest.java	69.30% <100.00%> (+0.30%)	⬆️
... and 8 more

... and 468 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testWriteRead
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testMultipleSnapshotAndRollback
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testList
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testIndicesDeletedFromRepository
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testDeleteBlobs
      1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.classMethod
      1 org.opensearch.cluster.ClusterHealthIT.testHealthOnClusterManagerFailover

• URL: https://build.ci.opensearch.org/job/gradle-check/19184/
• CommitID: 7a40bdc
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mingshl]

Thank you @tlfeng !