-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls #873
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small comments, rest LGTM
build.gradle
Outdated
@@ -158,7 +158,7 @@ dependencies { | |||
implementation group: 'org.apache.commons', name: 'commons-lang3', version: "${versions.commonslang}" | |||
implementation "org.antlr:antlr4-runtime:4.10.1" | |||
implementation "com.cronutils:cron-utils:9.1.6" | |||
api "org.opensearch:common-utils:${common_utils_version}@jar" | |||
api files("/Users/snistala/Documents/opensearch/common-utils/build/libs/common-utils-3.0.0.0-SNAPSHOT.jar") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accidental change
@@ -92,6 +94,7 @@ public void onResponse(GetMappingsResponse getMappingsResponse) { | |||
applyAliasMappings(getMappingsResponse.getMappings(), logType, aliasMappings, partial, new ActionListener<>() { | |||
@Override | |||
public void onResponse(Collection<CreateMappingResult> createMappingResponse) { | |||
log.error("PERF_DEBUG_SAP: Completed create mappings for {}", indexName); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should change this to debug and remove the PERF_DEBUG_SAP
prefix
@@ -110,6 +113,7 @@ public void onResponse(Collection<CreateMappingResult> createMappingResponse) { | |||
|
|||
@Override | |||
public void onFailure(Exception e) { | |||
log.error("PERF_DEBUG_SAP: Failed to create mappings for {}", indexName ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above, though this one could stay as error
@@ -220,19 +222,22 @@ private void checkIndicesAndExecute( | |||
ActionListener<IndexDetectorResponse> listener, | |||
User user | |||
) { | |||
log.error("PERF_DEBUG_SAP: check indices and execute began"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above and throughout this file
…pdation Signed-off-by: Surya Sashank Nistala <[email protected]>
…flow Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Surya Sashank Nistala <[email protected]>
3dc8d37
to
56c07ea
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #873 +/- ##
============================================
- Coverage 24.79% 24.63% -0.16%
- Complexity 1026 1030 +4
============================================
Files 277 277
Lines 12702 12817 +115
Branches 1394 1393 -1
============================================
+ Hits 3149 3158 +9
- Misses 9288 9395 +107
+ Partials 265 264 -1 ☔ View full report in Codecov by Sentry. |
@@ -2056,7 +2056,7 @@ public void testCreateDetectorWithCloudtrailAggrRuleWithEcsFields() throws IOExc | |||
// both req params and req body are supported | |||
createMappingRequest.setJsonEntity( | |||
"{\n" + | |||
" \"index_name\": \"" + index + "\",\n" + | |||
" \"index_name\": \"cloudtrail\",\n" + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should still be a variable, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
its a test
@@ -1,322 +0,0 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the reason to delete this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
twas unintentional.
reverted.
thanks for flagging this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we rename the PR description to Pass rule field names in doc level queries during monitor update/creation.
?
Signed-off-by: Surya Sashank Nistala <[email protected]>
The blocking call removals is also a key fix as that has caused create/update API to get stuck without response. I would prefer retaining it. Open to discuss further if you feel otherwise |
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-873-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 887739066de2d636d008711f12eec226d2b935b1
# Push it to GitHub
git push --set-upstream origin backport/backport-873-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x Then, create a pull request where the |
…emove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]>
…emove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]>
…emove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]>
* support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]>
Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 4d4f5e3) Co-authored-by: Joanne Wang <[email protected]> Reduce log level for informative message (opensearch-project#203) (opensearch-project#833) Signed-off-by: Enrico Tröger <[email protected]> Co-authored-by: Enrico Tröger <[email protected]> Updated alert creation following common-utils PR 584. (opensearch-project#837) (opensearch-project#839) Signed-off-by: AWSHurneyt <[email protected]> (cherry picked from commit 8adb9c3) Co-authored-by: AWSHurneyt <[email protected]> Release notes for 2.12.0 (opensearch-project#834) (opensearch-project#841) * release notes for 2.12 Signed-off-by: Joanne Wang <[email protected]> * update release notes Signed-off-by: Joanne Wang <[email protected]> * update release notes Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 414484a) Co-authored-by: Joanne Wang <[email protected]> Remove blocking calls and change threat intel feed flow to event driven (opensearch-project#871) (opensearch-project#876) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <[email protected]> * fix javadocs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert try catch removals Signed-off-by: Surya Sashank Nistala <[email protected]> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <[email protected]> * add try catch Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> (cherry picked from commit 172d58d) Co-authored-by: Surya Sashank Nistala <[email protected]> Fail the flow the when detectot type is missing in the log types index (opensearch-project#845) (opensearch-project#857) Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 8d19912) Co-authored-by: Megha Goyal <[email protected]> [BUG] ArrayIndexOutOfBoundsException for inconsistent detector index behavior (opensearch-project#843) (opensearch-project#858) * Catch ArrayIndexOutOfBoundsException when detector is missing Signed-off-by: Megha Goyal <[email protected]> * Add a check on SearchHits.getHits() length Signed-off-by: Megha Goyal <[email protected]> * Remove index out of bounds exception Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 0ef8543) Co-authored-by: Megha Goyal <[email protected]> Backport opensearch-project#873 and opensearch-project#789 (opensearch-project#895) * support object fields in aggregation based sigma rules (opensearch-project#789) Signed-off-by: Subhobrata Dey <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (opensearch-project#786) (opensearch-project#788) (opensearch-project#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <[email protected]> Add throw for empty strings in rules with modifier contains, startwith, and endswith (opensearch-project#860) (opensearch-project#896) * add validation for empty strings with contains, startswith and endswith modifiers * throw exception if empty string with contains, startswith, or endswith * change var name * add modifiers to log --------- Signed-off-by: Joanne Wang <[email protected]> Add an "exists" check for "not" condition in sigma rules (opensearch-project#852) (opensearch-project#897) * test design Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * cleaning up Signed-off-by: Joanne Wang <[email protected]> * testing Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * working version Signed-off-by: Joanne Wang <[email protected]> * refactored querybackend Signed-off-by: Joanne Wang <[email protected]> * working on tests Signed-off-by: Joanne Wang <[email protected]> * fixed alerting and finding tests Signed-off-by: Joanne Wang <[email protected]> * fix correlation tests Signed-off-by: Joanne Wang <[email protected]> * working all tests Signed-off-by: Joanne Wang <[email protected]> * moved test and changed alias for adldap Signed-off-by: Joanne Wang <[email protected]> * added more tests Signed-off-by: Joanne Wang <[email protected]> * cleanup code Signed-off-by: Joanne Wang <[email protected]> * remove exists flag Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> (cherry picked from commit 656a5fe) Co-authored-by: Joanne Wang <[email protected]> Add goyamegh as a maintainer (opensearch-project#868) (opensearch-project#899) Signed-off-by: Megha Goyal <[email protected]> Refactor invocation of Action listeners in correlations (opensearch-project#880) (opensearch-project#900) * Refactor invocation of Action listeners in correlations * Close hanging tasks in correlations workflow * Logging finding id and monitor id in error logs --------- Signed-off-by: Megha Goyal <[email protected]> Add search request timeouts for correlations workflows (opensearch-project#893) (opensearch-project#901) * Reinstating more leaks plugged-in for correlations workflows Signed-off-by: Megha Goyal <[email protected]> * Add search timeouts to all correlation searches Signed-off-by: Megha Goyal <[email protected]> * Fix logging and exception messages Signed-off-by: Megha Goyal <[email protected]> * Change search timeout to 30 seconds Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> (cherry picked from commit 75c4429) Co-authored-by: Megha Goyal <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>
…emove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]>
* query_field_names bugfix (#335) Signed-off-by: Petar Dzepina <[email protected]> * Reduce log level for informative message (#203) Signed-off-by: Enrico Tröger <[email protected]> * fix detector writeTo() method missing fields (#695) * fix detector writeTo() method missing fields Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * fix null query filter conversion from sigma to query string query (#722) * fix null query filter conversion from sigma to query string query Signed-off-by: Surya Sashank Nistala <[email protected]> * fix rule to query conversion tests for null filter Signed-off-by: Surya Sashank Nistala <[email protected]> * enhance test to verify non null doc doesnt match null query Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * Bump version 2.5.1, fix build Signed-off-by: Chase Engelbrecht <[email protected]> * Manual cherry-pick of #873 Signed-off-by: Chase Engelbrecht <[email protected]> * Fix miss from manual cherry-pick Signed-off-by: Chase Engelbrecht <[email protected]> * Undo exceptional case not originally present Signed-off-by: Chase Engelbrecht <[email protected]> * Revert test from previous commit reversion Signed-off-by: Chase Engelbrecht <[email protected]> * Fix another miss from manual cherry-pick Signed-off-by: Chase Engelbrecht <[email protected]> * Revert behavior for no rules in detector Signed-off-by: Chase Engelbrecht <[email protected]> --------- Signed-off-by: Petar Dzepina <[email protected]> Signed-off-by: Enrico Tröger <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Chase Engelbrecht <[email protected]> Co-authored-by: Petar Dzepina <[email protected]> Co-authored-by: Enrico Tröger <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]>
…emove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]>
* Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <[email protected]> * add integ test Signed-off-by: Joanne Wang <[email protected]> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <[email protected]> * add comments Signed-off-by: Joanne Wang <[email protected]> * fix integ tests Signed-off-by: Joanne Wang <[email protected]> * moved logic to method for better readability Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * version bump Signed-off-by: Surya Sashank Nistala <[email protected]> * AggregationBackendTests fix Signed-off-by: Riya Saxena <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Riya Saxena <[email protected]> Co-authored-by: Joanne Wang <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]>
* Version bump to 2.7.1 Signed-off-by: Megha Goyal <[email protected]> * fix detector writeTo() method missing fields (#695) * fix detector writeTo() method missing fields Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * fix null query filter conversion from sigma to query string query (#722) * fix null query filter conversion from sigma to query string query Signed-off-by: Surya Sashank Nistala <[email protected]> * fix rule to query conversion tests for null filter Signed-off-by: Surya Sashank Nistala <[email protected]> * enhance test to verify non null doc doesnt match null query Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * * #709 Return empty response for empty mappings and no applied aliases Signed-off-by: Megha Goyal <[email protected]> * Adding integ tests for empty mappings/aliases use-cases Signed-off-by: Megha Goyal <[email protected]> * Fix unit tests for MappingsTraverser Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> * fixing build failures for previous cherry-pick PR for SAP2.7 (to be squashed) Signed-off-by: Dennis Toepker <[email protected]> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <[email protected]> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <[email protected]> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * add finding mappings Signed-off-by: Surya Sashank Nistala <[email protected]> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <[email protected]> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * fixing an import Signed-off-by: Dennis Toepker <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Dennis Toepker <[email protected]> Co-authored-by: Megha Goyal <[email protected]> Co-authored-by: Surya Sashank Nistala <[email protected]> Co-authored-by: Megha Goyal <[email protected]> Co-authored-by: Dennis Toepker <[email protected]>
Description
Pass rule field names in doc level queries during monitor/creation. This utilizes optimization done in alerting to avoid fetching entire _source of docs in doc level monitor
Issues Resolved
opensearch-project/alerting#1367
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.