Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk index findings and sequentially invoke auto-correlations #1355

Merged
merged 6 commits into from
Feb 6, 2024

Conversation

goyamegh
Copy link
Collaborator

@goyamegh goyamegh commented Dec 27, 2023

Issue #, if available:
#1333

Description of changes:
All the findings once generated, are indexed one-by-one today. This PR attempts to optimize this by making a bulk index request for all the generated findings. Once completed, each finding will then be published to kick-start auto-correlations subsequently.

CheckList:

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@goyamegh goyamegh force-pushed the main-bulkIndexFindings branch from da330a8 to 03b86ae Compare February 1, 2024 19:23
}
}

val actionCtx = triggerCtx.copy(
triggeredDocs = triggerResult.triggeredDocs,
relatedFindings = findings,
// confirm if this is right or only trigger-able findings should be present in this list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: plz add TODO/FIXME and maintain pr in draft if it's not ready to merge

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the previous behaviour? why are we not just refactoring? is there a behaviour change?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is preserving the existing behavior. Left by mistake, removed the comment.

val FINDINGS_INDEXING_BATCH_SIZE = Setting.intSetting(
"plugins.alerting.alert_findings_indexing_batch_size",
DEFAULT_FINDINGS_INDEXING_BATCH_SIZE,
0,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

min cant be 0

@@ -346,7 +346,8 @@ internal class AlertingPlugin : PainlessExtension, ActionPlugin, ScriptPlugin, R
AlertingSettings.FINDING_HISTORY_MAX_DOCS,
AlertingSettings.FINDING_HISTORY_INDEX_MAX_AGE,
AlertingSettings.FINDING_HISTORY_ROLLOVER_PERIOD,
AlertingSettings.FINDING_HISTORY_RETENTION_PERIOD
AlertingSettings.FINDING_HISTORY_RETENTION_PERIOD,
AlertingSettings.FINDINGS_INDEXING_BATCH_SIZE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:BULK

val findingDocPairs = mutableListOf<Pair<String, String>>()
val findings = mutableListOf<Finding>()
val indexRequests = mutableListOf<IndexRequest>()
monitorCtx.findingsIndexBatchSize = FINDINGS_INDEXING_BATCH_SIZE.get(monitorCtx.settings)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this - monitorCtx.findingsIndexBatchSize??

Why are we mutating value of monitorCtx object?? that object should only be read from

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent is to update this value in alerting, when it is updated via _cluster/settings.

@eirsep
Copy link
Member

eirsep commented Feb 2, 2024

can we test the new setting?

val findings = mutableListOf<Finding>()
val indexRequests = mutableListOf<IndexRequest>()
monitorCtx.findingsIndexBatchSize = FINDINGS_INDEXING_BATCH_SIZE.get(monitorCtx.settings)
monitorCtx.clusterService!!.clusterSettings.addSettingsUpdateConsumer(FINDINGS_INDEXING_BATCH_SIZE) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we aren't supposed to add settings consumer in the middle of monitor execution. plz do at node startup

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to registerConsumers() in MonitorRunnerService.

) {
if (indexRequests.isNotEmpty()) {
val bulkResponse: BulkResponse = monitorCtx.client!!.suspendUntil {
bulk(BulkRequest().add(indexRequests).setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE), it)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont refresh for every batch. refresh findings index only once after all batches are bulk-ingested

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Signed-off-by: Megha Goyal <[email protected]>
@goyamegh goyamegh force-pushed the main-bulkIndexFindings branch from 4ec7848 to ae32748 Compare February 5, 2024 18:33
Copy link
Member

@eirsep eirsep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz add a test where you make bulk finding size =2 and simulate creation of 10 findings

@eirsep
Copy link
Member

eirsep commented Feb 6, 2024

fix ktlint
CIs are failing.

@eirsep eirsep self-requested a review February 6, 2024 19:07
Signed-off-by: Megha Goyal <[email protected]>
@goyamegh goyamegh force-pushed the main-bulkIndexFindings branch from 64e1f20 to 3bd7888 Compare February 6, 2024 19:16
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.9 2.9
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.9
# Create a new branch
git switch --create backport-1355-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.9
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport-1355-to-2.9.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.8 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.8 2.8
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.8
# Create a new branch
git switch --create backport-1355-to-2.8
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.8
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.8

Then, create a pull request where the base branch is 2.8 and the compare/head branch is backport-1355-to-2.8.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.7 2.7
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.7
# Create a new branch
git switch --create backport-1355-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.7
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport-1355-to-2.7.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.6 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.6 2.6
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.6
# Create a new branch
git switch --create backport-1355-to-2.6
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.6
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.6

Then, create a pull request where the base branch is 2.6 and the compare/head branch is backport-1355-to-2.6.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.5 2.5
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.5
# Create a new branch
git switch --create backport-1355-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport-1355-to-2.5.

opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 11, 2024
* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
(cherry picked from commit b561965)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 11, 2024
* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
(cherry picked from commit b561965)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.11
# Create a new branch
git switch --create backport-1355-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport-1355-to-2.11.

eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 13, 2024
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
jowg-amazon pushed a commit that referenced this pull request Mar 14, 2024
* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
goyamegh added a commit to goyamegh/alerting that referenced this pull request Mar 14, 2024
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
jowg-amazon added a commit that referenced this pull request Mar 14, 2024
…#1471)

* Bulk index findings and sequentially invoke auto-correlations (#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix integTests

Signed-off-by: Joanne Wang <[email protected]>

* clean up doc level queries on dry run (#1430)

Signed-off-by: Joanne Wang <[email protected]>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix tests

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add javadoc

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix integ tests again

Signed-off-by: Joanne Wang <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Joanne Wang <[email protected]>
Co-authored-by: Megha Goyal <[email protected]>
Co-authored-by: Surya Sashank Nistala <[email protected]>
eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 14, 2024
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
eirsep added a commit that referenced this pull request Mar 14, 2024
…, #1441 to 2.9 (#1469)

* optimize doc-level monitor execution workflow for datastreams (#1302)

* optimize doc-level monitor execution for datastreams

Signed-off-by: Subhobrata Dey <[email protected]>

* add more tests to address comments

Signed-off-by: Subhobrata Dey <[email protected]>

* add integTest for multiple datastreams inside a single index pattern

* add integTest for multiple datastreams inside a single index pattern

Signed-off-by: Subhobrata Dey <[email protected]>

---------

Signed-off-by: Subhobrata Dey <[email protected]>

* Bulk index findings and sequentially invoke auto-correlations (#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>

* fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* clean up doc level queries on dry run (#1430)

Signed-off-by: Joanne Wang <[email protected]>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix tests

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add javadoc

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add distributed locking to jobs in alerting (#1403)

Signed-off-by: Subhobrata Dey <[email protected]>

* 2.9.1 version bump

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix compilation issues

Signed-off-by: Surya Sashank Nistala <[email protected]>

* dummy commit

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix findings index schema version tsts

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Subhobrata Dey <[email protected]>
Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Joanne Wang <[email protected]>
Co-authored-by: Subhobrata Dey <[email protected]>
Co-authored-by: Megha Goyal <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Joanne Wang <[email protected]>
engechas pushed a commit to engechas/alerting that referenced this pull request Mar 18, 2024
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
@engechas engechas mentioned this pull request Mar 18, 2024
1 task
engechas pushed a commit to engechas/alerting that referenced this pull request Mar 18, 2024
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>
engechas added a commit that referenced this pull request Mar 18, 2024
* log error messages and clean up monitor when indexing doc level queries or metadata creation fails (#900)

* log errors and clean up monitor when indexing doc level queries or metadata creation fails
* refactor delete monitor action to re-use delete methods
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* optimize doc-level monitor workflow for index patterns (#1097)

Signed-off-by: Subhobrata Dey <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* optimize doc-level monitor execution workflow for datastreams (#1302)

* optimize doc-level monitor execution for datastreams

Signed-off-by: Subhobrata Dey <[email protected]>

* add more tests to address comments

Signed-off-by: Subhobrata Dey <[email protected]>

* add integTest for multiple datastreams inside a single index pattern

* add integTest for multiple datastreams inside a single index pattern

Signed-off-by: Subhobrata Dey <[email protected]>

---------

Signed-off-by: Subhobrata Dey <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* Bulk index findings and sequentially invoke auto-correlations (#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix tests

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add javadoc

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix BWC tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* clean up doc level queries on dry run (#1430)

Signed-off-by: Joanne Wang <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix import

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix tests

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix BWC version

Signed-off-by: Chase Engelbrecht <[email protected]>

* Fix another test

Signed-off-by: Chase Engelbrecht <[email protected]>

* Revert order of operations change

Signed-off-by: Chase Engelbrecht <[email protected]>

---------

Signed-off-by: Subhobrata Dey <[email protected]>
Signed-off-by: Chase Engelbrecht <[email protected]>
Signed-off-by: Megha Goyal <[email protected]>
Signed-off-by: Surya Sashank Nistala <[email protected]>
Signed-off-by: Joanne Wang <[email protected]>
Co-authored-by: Surya Sashank Nistala <[email protected]>
Co-authored-by: Subhobrata Dey <[email protected]>
Co-authored-by: Megha Goyal <[email protected]>
Co-authored-by: Joanne Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants