forked from opensearch-project/anomaly-detection
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix inference logic and standardize config index mapping
This commit addresses several issues and improvements in the inference logic and config index mapping: 1. Fixes in RealTimeInferencer: * Previously, we checked if the last update time of the model state was within the current interval and skipped inference if it was. However, this led to excessive skipping of inference because the last update time was updated when retrieving the model state from the cache. * Introduced lastSeenExecutionEndTime in the model state, which specifically tracks the last time a sample was processed during inference (not training). This ensures more accurate control over when inference should be skipped. 2. Consistent Naming in Config Index Mapping: * To maintain consistency across the codebase, changed defaultFill to default_fill in the Config index mapping, following the underscore naming convention used elsewhere. 3. Additional Null Checks: * Added more null checks for the defaultFill field in the Config constructor to improve robustness. Testing: * Added a smoke test to allow the job scheduler to trigger anomaly detection inferencing, successfully reproducing and verifying the fix for item #1.* added unit tests for item #3. Signed-off-by: Kaituo Li <[email protected]>
- Loading branch information
Showing
17 changed files
with
314 additions
and
67 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
name: Run long running tests | ||
on: | ||
push: | ||
branches: | ||
- "*" | ||
pull_request: | ||
branches: | ||
- "*" | ||
|
||
env: | ||
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true | ||
|
||
jobs: | ||
Get-CI-Image-Tag: | ||
uses: opensearch-project/opensearch-build/.github/workflows/get-ci-image-tag.yml@main | ||
with: | ||
product: opensearch | ||
|
||
Run-Tests: | ||
needs: Get-CI-Image-Tag | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
# each test scenario (rule, hc, single_stream) is treated as a separate job. | ||
test: [smoke] | ||
fail-fast: false | ||
concurrency: | ||
# The concurrency setting is used to limit the concurrency of each test scenario group to ensure they do not run concurrently on the same machine. | ||
group: ${{ github.workflow }}-${{ matrix.test }} | ||
name: Run long running tests | ||
|
||
container: | ||
# using the same image which is used by opensearch-build team to build the OpenSearch Distribution | ||
# this image tag is subject to change as more dependencies and updates will arrive over time | ||
image: ${{ needs.Get-CI-Image-Tag.outputs.ci-image-version-linux }} | ||
# need to switch to root so that github actions can install runner binary on container without permission issues. | ||
options: --user root | ||
|
||
steps: | ||
- name: Setup Java | ||
uses: actions/setup-java@v3 | ||
with: | ||
distribution: 'temurin' | ||
java-version: 21 | ||
|
||
- name: Checkout AD | ||
uses: actions/checkout@v3 | ||
|
||
- name: Build and Run Tests | ||
run: | | ||
chown -R 1000:1000 `pwd` | ||
case ${{ matrix.test }} in | ||
smoke) | ||
su `id -un 1000` -c "./gradlew integTest --tests 'org.opensearch.ad.e2e.SingleStreamSmokeIT' \ | ||
-Dtests.seed=B4BA12CCF1D9E825 -Dtests.security.manager=false \ | ||
-Dtests.jvm.argline='-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m' \ | ||
-Dtests.locale=ar-JO -Dtests.timezone=Asia/Samarkand -Dlong-running=true \ | ||
-Dtests.timeoutSuite=3600000! -Dtest.logs=true" | ||
;; | ||
esac |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.