Refinement of Forecasting and AD Precision/Recall Improvements

[kaituo]

This PR addresses several improvements related to forecasting and anomaly detection (AD) precision/recall. It introduces changes to accommodate forecasting functionality, which is currently disabled as it's not yet released. Additionally, it reverts name changes introduced in a previous PR #1173 due to the unreleased status of forecasting.

Changes Made:

• Integration of forecasting-related improvements.
• Reversion of name changes for compatibility reasons.
• Introduce rule based AD.

Testing Done:

• Verified frontend workflow remains functional: creation, previewing, historical, and real-time detection.
• All existing unit and integration tests pass successfully.
• Added a new integration test (RuleModelPerfIT) to validate rule-based AD improvements in precision/recall.

Next Steps:

• Add new AD tests before the 2.15 release to meet coverage requirements.
• Conduct backward compatibility tests to ensure compatibility with existing functionality.

The merge was attempted earlier to prevent blocking other teammates from submitting their changes.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[kaituo]

WhiteSource Security Check failed because the requests library version is lower than 2.32. This dependency is brought in by opensearch_py in /dataGeneration/requirements.txt. Despite updating our requirements.txt file to specify the desired version of requests as requests>=2.32.0, the security check still fails.

The limitation on the version of requests to be higher than 2.31 is not set in the opensearch_py project's dev-requirements.txt file (https://github.com/opensearch-project/opensearch-py/blob/2.0/dev-requirements.txt#L1).

As this issue pertains to testing and is not utilized in production, I will defer addressing it for now.

[kaituo]

BWC test failed as opensearch-anomaly-detection-2.15.0.0.zip is not available:

Can't get https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.15.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-anomaly-detection-2.15.0.0.zip to /__w/anomaly-detection/anomaly-detection/src/test/resources/org/opensearch/ad/bwc/anomaly-detection/2.15.0.0/opensearch-anomaly-detection-2.15.0.0.zip

After backporting to 2.x, I will bump the version to 2.15 and rerun the CI.

[jackiehanyang]

could getRules() returns an empty list? should we also handle that to avoid unnecessary nested loops when there are no rules?

[jackiehanyang]

went through this pr together with Kaituo offline as it's a huge one. Overall LGTM, most code changes are just refactoring and moving between packages

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/anomaly-detection/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/anomaly-detection/backport-2.x
# Create a new branch
git switch --create backport/backport-1210-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8141c27b3c2039f4fe239f9d6480b923c441f23f
# Push it to GitHub
git push --set-upstream origin backport/backport-1210-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/anomaly-detection/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1210-to-2.x.