[Security Solution][Detection Engine] adds alert suppression for New Terms rule type #178294

vitaliidm · 2024-03-08T11:22:46Z

Summary

addresses https://github.com/elastic/security-team/issues/8824
adds alert suppression for new terms rule type
fixes getOpenAlerts test function, which returned closed alerts as well

UI

Checklist

Functional changes are hidden behind a feature flag

Feature flag alertSuppressionForNewTermsRuleEnabled
Functional changes are covered with a test plan and automated tests.

Test plan: https://github.com/elastic/security-team/pull/9045
Stability of new and changed tests is verified using the Flaky Test Runner.

Cypress ESS: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5547
Cypress Serverless: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5548

FTR ESS: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5596
FTR Serverless: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5597

Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless.
Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders.

Existing AlertSuppression schema field is used for New terms rule, the one that used for Query and IM rules.

    alert_suppression:
      $ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression'

where

    AlertSuppression:
      type: object
      properties:
        group_by:
          $ref: '#/components/schemas/AlertSuppressionGroupBy'
        duration:
          $ref: '#/components/schemas/AlertSuppressionDuration'
        missing_fields_strategy:
          $ref: '#/components/schemas/AlertSuppressionMissingFieldsStrategy'
      required:
        - group_by

Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both).

elastic/security-docs#5030

…Terms rule type

…taliidm/kibana into de_8_14/new_terms_suppression

elasticmachine · 2024-04-02T15:47:23Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

vitaliidm · 2024-04-02T15:48:13Z

Files by Code Owner

elastic/obs-ux-management-team

x-pack/plugins/rule_registry/server/utils/create_persistence_rule_type_wrapper.ts
x-pack/plugins/rule_registry/server/utils/persistence_types.ts

elastic/response-ops

x-pack/plugins/alerting/server/integration_tests/snapshots/serverless_upgrade_and_rollback_checks.test.ts.snap
x-pack/plugins/rule_registry/server/utils/create_persistence_rule_type_wrapper.ts
x-pack/plugins/rule_registry/server/utils/persistence_types.ts

elastic/security-detection-engine

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_request_schema.test.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/description_step/index.test.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/schema.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/use_experimental_feature_fields_transform.ts
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/pages/rule_creation/helpers.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema/model/rule_schemas.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/bulk_create_suppressed_alerts_in_memory.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/multi_terms_composite.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/types.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/wrap_suppressed_new_terms_alerts.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/wrap_suppressed_new_terms_alerts.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/bulk_create_suppressed_alerts_in_memory.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/bulk_create_with_suppression.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/get_is_alert_suppression_active.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/index.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/partition_missing_fields_events.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/search_after_bulk_create_suppressed_alerts.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/suppression_utils.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/suppression_utils.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/wrap_suppressed_alerts.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/workflows/trial_license_complete_tier/role_based_rule_exceptions_workflows.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/configs/serverless.config.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/eql.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/esql.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/index.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/machine_learning.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/new_terms.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/new_terms_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/query.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/saved_query.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threat_match.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threat_match_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threshold.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threshold_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/timestamps.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_suppression_serverless_essentials.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_supression_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/indicator_match_rule_suppression_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/new_terms_rule.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/threshold_rule_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/threshold_rule_serverless_essentials.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_edit/new_terms_rule.cy.ts

elastic/security-detection-rule-management

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_request_schema.test.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml
x-pack/plugins/security_solution/public/detection_engine/rule_management/components/rule_details/rule_definition_section.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.test.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.tsx
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema/model/rule_schemas.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/prebuilt_rules_preview.cy.ts

elastic/security-detections-response

x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/get_alerts.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/get_opened_alerts.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/index.ts

elastic/security-engineering-productivity

x-pack/test/security_solution_cypress/config.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_suppression_serverless_essentials.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/common_flows_supression_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/indicator_match_rule_suppression_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/new_terms_rule.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/threshold_rule_ess_basic.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_creation/threshold_rule_serverless_essentials.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/rule_edit/new_terms_rule.cy.ts
x-pack/test/security_solution_cypress/cypress/e2e/detection_response/rule_management/prebuilt_rules/prebuilt_rules_preview.cy.ts
x-pack/test/security_solution_cypress/cypress/tasks/create_new_rule.ts
x-pack/test/security_solution_cypress/serverless_config.ts

elastic/security-solution

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_request_schema.test.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts
x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml
x-pack/plugins/security_solution/common/detection_engine/constants.ts
x-pack/plugins/security_solution/common/detection_engine/utils.test.ts
x-pack/plugins/security_solution/common/experimental_features.ts
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/description_step/index.test.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/schema.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/step_define_rule/use_experimental_feature_fields_transform.ts
x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/pages/rule_creation/helpers.ts
x-pack/plugins/security_solution/public/detection_engine/rule_management/components/rule_details/rule_definition_section.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.test.tsx
x-pack/plugins/security_solution/public/detection_engine/rule_management/logic/use_alert_suppression.tsx
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/normalization/rule_converters.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema/model/rule_schemas.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/bulk_create_suppressed_alerts_in_memory.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/create_new_terms_alert_type.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/multi_terms_composite.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/types.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/wrap_suppressed_new_terms_alerts.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/new_terms/wrap_suppressed_new_terms_alerts.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/bulk_create_suppressed_alerts_in_memory.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/bulk_create_with_suppression.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/get_is_alert_suppression_active.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/index.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/partition_missing_fields_events.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/search_after_bulk_create_suppressed_alerts.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/suppression_utils.test.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/suppression_utils.ts
x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/wrap_suppressed_alerts.ts
x-pack/test/security_solution_api_integration/config/ess/config.base.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/exceptions/workflows/trial_license_complete_tier/role_based_rule_exceptions_workflows.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/configs/serverless.config.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/eql.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/esql.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/index.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/machine_learning.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/new_terms.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/new_terms_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/query.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/saved_query.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threat_match.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threat_match_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threshold.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/threshold_alert_suppression.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/timestamps.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/get_alerts.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/get_opened_alerts.ts
x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/alerts/index.ts

pmuellr

LGTM

...2e/detection_response/detection_engine/rule_creation/common_flows_supression_ess_basic.cy.ts

MadameSheema

security-engineering-productivity changes LGTM, even so, please take into consideration the following:

As the FF has been enabled in x-pack/test/security_solution_cypress/config.ts all the tests executed for both ESS and Serverless on PRs will be affected by that change, not only the ones were are specifically set at the top of the spec file.
On MKI environments, we don't have the ability to enable FF through the automation, once the changes asked regarding tagging are implemented, all the serverless tests that have the feature flag will fail on the periodic pipeline. cc @yctercero
I saw you are doing changes as well in archives living in x-pack/test/functional/es_archives/security_solution , now we are able to reuse archives on that location in our Cypress tests as well. Letting you know in case that this new functionality might facilitate your work.

maximpn

Rule Management Area LGTM 👍

I checked the PR locally in ESS and Serverless for one field alert suppression and haven't found any problems.

rylnd

I gave a first pass here and everything looks GREAT. Pretty straightforward overall, with some reasonable refactorings that didn't explode the diff too much.

I had a few nits, but they were mainly around documentation/posterity stuff.

I have NOT reviewed the integration/cypress tests yet. I plan to go through those alongside the EQL suppression branch to compare/contrast coverage and approach. If I have any suggestions for adding coverage, we can address those after this gets merged.

Between the plentiful tests, the green build, and the feature flag, I'm approving this as is. I did check this out locally, tried a few different suppression scenarios, tried to break the rule creation form, etc. Everything worked for me!

.../public/detection_engine/rule_management/components/rule_details/rule_definition_section.tsx

rylnd · 2024-04-05T20:20:29Z

...curity_solution/public/detection_engine/rule_management/logic/use_alert_suppression.test.tsx


 describe('useAlertSuppression', () => {
+  it('should return isSuppressionEnabled false if rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {


Nit: the should is unnecessary here. I'm going to die on this hill 😅 .

Suggested change

it('should return isSuppressionEnabled false if rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {

it('returns isSuppressionEnabled: false when rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {

a lot of other tests have should, which is used widely across app.
What the criteria to use/not use should in your opinion?
Maybe based on it, we can draw some guideline on its usage?

It's not a big deal, but I have never encountered a situation where the addition of "should" added any clarity to a test's description.

.../server/lib/detection_engine/rule_types/new_terms/bulk_create_suppressed_alerts_in_memory.ts

rylnd · 2024-04-05T21:12:13Z

.../server/lib/detection_engine/rule_types/new_terms/bulk_create_suppressed_alerts_in_memory.ts

+  experimentalFeatures: ExperimentalFeatures | undefined;
+}
+/**
+ * bulk create and suppress alerts in memory,


Nit: these comments don't provide much more information than the name does. I would mention that this is explicitly for new terms alerts, or at least deals with EventsAndTerms[] instead of normal alerts.

The note about missing fields logic is nice, but it would be nice to discuss it more explicitly, i.e. "if {@param alertSuppression.missingFieldsStrategy} is X, alerts missing the suppression field(s) will not be suppressed."

But: typing out the above, this doesn't feel like the right place to document that behavior. I would capture it in documentation, and ideally a unit test for this function, otherwise an integration test of some kind.

I added a bit more details to description, mentioned difference with common utility.

There multiple FTR tests that cover missing fields logic:

kibana/x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/trial_license_complete_tier/execution_logic/new_terms_alert_suppression.ts

Lines 874 to 981 in ff8a41f

it('should not suppress alerts with missing fields if configured so', async () => {

const id = uuidv4();

const firstTimestamp = '2020-10-28T05:45:00.000Z';

const secondTimestamp = '2020-10-28T06:10:00.000Z';

const historicalDocuments = [

{

host: { name: 'host-a', ip: '127.0.0.1' },

'agent.version': 1,

id,

'@timestamp': historicalWindowStart,

},

];

const firstExecutionDocuments = [

{

id,

'@timestamp': firstTimestamp,

host: { name: 'host-a', ip: '127.0.0.3' },

},

{

id,

'@timestamp': firstTimestamp,

host: { name: 'host-a', ip: '127.0.0.4' },

},

{

id,

'@timestamp': firstTimestamp,

host: { ip: '127.0.0.5' }, // doc 1 with missing host.name field

},

{

id,

'@timestamp': firstTimestamp,

host: { ip: '127.0.0.6' }, // doc 2 with missing host.name field

},

];

const secondExecutionDocuments = [

{

host: { name: 'host-a', ip: '127.0.0.10' },

id,

'@timestamp': secondTimestamp,

},

{

host: { ip: '127.0.0.11' }, // doc 3 with missing host.name field

id,

'@timestamp': secondTimestamp,

},

];

await indexListOfDocuments([

...historicalDocuments,

...firstExecutionDocuments,

...secondExecutionDocuments,

]);

const rule: NewTermsRuleCreateProps = {

...getCreateNewTermsRulesSchemaMock('rule-1', true),

new_terms_fields: ['host.ip'],

query: `id: "${id}"`,

index: ['ecs_compliant'],

history_window_start: historicalWindowStart,

alert_suppression: {

group_by: ['host.name'],

duration: {

value: 300,

unit: 'm',

},

missing_fields_strategy: 'doNotSuppress',

},

from: 'now-35m',

interval: '30m',

};

const { previewId } = await previewRule({

supertest,

rule,

timeframeEnd: new Date('2020-10-28T06:30:00.000Z'),

invocationCount: 2,

});

const previewAlerts = await getPreviewAlerts({

es,

previewId,

sort: ['host.name', ALERT_ORIGINAL_TIME],

});

expect(previewAlerts.length).toEqual(4);

expect(previewAlerts[0]._source).toEqual({

...previewAlerts[0]._source,

[ALERT_SUPPRESSION_TERMS]: [

{

field: 'host.name',

value: ['host-a'],

},

],

[ALERT_ORIGINAL_TIME]: firstTimestamp,

[ALERT_SUPPRESSION_START]: firstTimestamp,

[ALERT_SUPPRESSION_END]: secondTimestamp,

[ALERT_SUPPRESSION_DOCS_COUNT]: 2,

});

// rest of alerts are not suppressed and do not have suppress properties

previewAlerts.slice(1).forEach((previewAlert) => {

const source = previewAlert._source;

expect(source).toHaveProperty('id', id);

expect(source).not.toHaveProperty(ALERT_SUPPRESSION_DOCS_COUNT);

expect(source).not.toHaveProperty(ALERT_SUPPRESSION_END);

expect(source).not.toHaveProperty(ALERT_SUPPRESSION_TERMS);

expect(source).not.toHaveProperty(ALERT_SUPPRESSION_DOCS_COUNT);

});

});

...ity_solution/server/lib/detection_engine/rule_types/utils/get_is_alert_suppression_active.ts

...ins/security_solution/server/lib/detection_engine/rule_types/utils/suppression_utils.test.ts

...ins/security_solution/server/lib/detection_engine/rule_types/utils/wrap_suppressed_alerts.ts

x-pack/test/functional/es_archives/security_solution/ecs_compliant/mappings.json

...security_solution_api_integration/test_suites/detections_response/utils/alerts/get_alerts.ts

...y_solution_api_integration/test_suites/detections_response/utils/alerts/get_opened_alerts.ts

vitaliidm · 2024-04-08T10:42:46Z

I gave a first pass here and everything looks GREAT. Pretty straightforward overall, with some reasonable refactorings that didn't explode the diff too much.
I had a few nits, but they were mainly around documentation/posterity stuff.
I have NOT reviewed the integration/cypress tests yet. I plan to go through those alongside the EQL suppression branch to compare/contrast coverage and approach. If I have any suggestions for adding coverage, we can address those after this gets merged.
Between the plentiful tests, the green build, and the feature flag, I'm approving this as is. I did check this out locally, tried a few different suppression scenarios, tried to break the rule creation form, etc. Everything worked for me!

Thanks for review and feedback, @rylnd
I have addressed comments and once it merged we can go ahead with updating EQL PR and resolve possible conflicts

kibana-ci · 2024-04-08T18:17:10Z

💚 Build Succeeded

Buildkite Build
Commit: 2237d76

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`ruleRegistry`	243	245	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	17.0MB	17.0MB	+759.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	73.4KB	73.4KB	+54.0B

Unknown metric groups

API count

id	before	after	diff
`ruleRegistry`	272	274	+2

References to deprecated APIs

id	before	after	diff
`securitySolution`	533	534	+1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm

This was missed in a previous merge of #178294, but was somehow not caught by CI. Suspicious!

[Security Solution][Detection Engine] adds alert suppression for New …

12b5eff

…Terms rule type

vitaliidm self-assigned this Mar 8, 2024

vitaliidm added 23 commits March 8, 2024 11:27

add usint tests for converter

2444935

cypress tests

ec467bc

remove failures

68ef90c

fix tests

f7bd2fc

Merge branch 'main' into de_8_14/new_terms_suppression

4717113

Merge branch 'main' into de_8_14/new_terms_suppression

028efcc

add ftr tests

203cda5

split search and bulk create

b0a311f

add BE

0bd4ec8

Merge branch 'main' into de_8_14/new_terms_suppression

730f0cf

add more fixes

42770a9

fix tests

529733f

Merge branch 'de_8_14/new_terms_suppression' of https://github.com/vi…

04378e2

…taliidm/kibana into de_8_14/new_terms_suppression

Merge branch 'main' into de_8_14/new_terms_suppression

171b2fc

fixes some tests

97e43cc

Merge branch 'de_8_14/new_terms_suppression' of https://github.com/vi…

186ac3d

…taliidm/kibana into de_8_14/new_terms_suppression

fix lint

25c94b9

fix tests

a1a7496

Merge branch 'main' into de_8_14/new_terms_suppression

b10f6de

Merge branch 'main' into de_8_14/new_terms_suppression

46ce899

fixes

710dda5

refactoring/fixes

7061dbf

fix lints

890391e

vitaliidm added 2 commits April 3, 2024 14:02

Merge branch 'main' into de_8_14/new_terms_suppression

2995b0e

Merge branch 'main' into de_8_14/new_terms_suppression

bbfb3d4

pmuellr approved these changes Apr 4, 2024

View reviewed changes

MadameSheema reviewed Apr 5, 2024

View reviewed changes

...2e/detection_response/detection_engine/rule_creation/common_flows_supression_ess_basic.cy.ts Outdated Show resolved Hide resolved

MadameSheema approved these changes Apr 5, 2024

View reviewed changes

vitaliidm added 2 commits April 5, 2024 11:09

CR feedback

8280b6a

lint errors

9af1081

maximpn approved these changes Apr 5, 2024

View reviewed changes

vitaliidm added 4 commits April 5, 2024 15:46

Merge branch 'main' into de_8_14/new_terms_suppression

36e06a5

remove unnecessary file

08f8eb1

Merge branch 'main' into de_8_14/new_terms_suppression

5c36012

Merge branch 'main' into de_8_14/new_terms_suppression

d592c79

rylnd approved these changes Apr 5, 2024

View reviewed changes

vitaliidm added 5 commits April 8, 2024 10:14

Merge branch 'main' into de_8_14/new_terms_suppression

f3152bd

CR feedback

d47da36

CR: feedback

778b501

CR: add more test cases

ff8a41f

CR feedback

2f65849

vitaliidm added 3 commits April 8, 2024 11:47

fix jest test

d6200ff

Merge branch 'main' into de_8_14/new_terms_suppression

1d30c31

fix FTR test

2237d76

vitaliidm merged commit 52cfdd6 into elastic:main Apr 9, 2024
39 checks passed

kibanamachine added v8.14.0 backport:skip This commit does not require backporting labels Apr 9, 2024

rylnd added a commit that referenced this pull request Apr 10, 2024

Add missing argument to suppression function in eql executor

98541ab

This was missed in a previous merge of #178294, but was somehow not caught by CI. Suspicious!

Mikaayenson mentioned this pull request Aug 14, 2024

[FR] Add Alert Suppression for Addtional Rule Types elastic/detection-rules#3986

Merged

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] adds alert suppression for New Terms rule type #178294

[Security Solution][Detection Engine] adds alert suppression for New Terms rule type #178294

vitaliidm commented Mar 8, 2024 •

edited

Loading

elasticmachine commented Apr 2, 2024

vitaliidm commented Apr 2, 2024 •

edited by rylnd

Loading

Files by Code Owner

elastic/obs-ux-management-team

elastic/response-ops

elastic/security-detection-engine

elastic/security-detection-rule-management

elastic/security-detections-response

elastic/security-engineering-productivity

elastic/security-solution

pmuellr left a comment

MadameSheema left a comment

maximpn left a comment

rylnd left a comment

rylnd Apr 5, 2024

vitaliidm Apr 8, 2024

rylnd Apr 9, 2024

rylnd Apr 5, 2024

vitaliidm Apr 8, 2024

vitaliidm commented Apr 8, 2024

kibana-ci commented Apr 8, 2024

API count

References to deprecated APIs


		describe('useAlertSuppression', () => {
		it('should return isSuppressionEnabled false if rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {

	it('should return isSuppressionEnabled false if rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {
	it('returns isSuppressionEnabled: false when rule Type exists in SUPPRESSIBLE_ALERT_RULES and Feature Flag is disabled', () => {

	it('should not suppress alerts with missing fields if configured so', async () => {
	const id = uuidv4();
	const firstTimestamp = '2020-10-28T05:45:00.000Z';
	const secondTimestamp = '2020-10-28T06:10:00.000Z';
	const historicalDocuments = [
	{
	host: { name: 'host-a', ip: '127.0.0.1' },
	'agent.version': 1,
	id,
	'@timestamp': historicalWindowStart,
	},
	];

	const firstExecutionDocuments = [
	{
	id,
	'@timestamp': firstTimestamp,
	host: { name: 'host-a', ip: '127.0.0.3' },
	},
	{
	id,
	'@timestamp': firstTimestamp,
	host: { name: 'host-a', ip: '127.0.0.4' },
	},
	{
	id,
	'@timestamp': firstTimestamp,
	host: { ip: '127.0.0.5' }, // doc 1 with missing host.name field
	},
	{
	id,
	'@timestamp': firstTimestamp,
	host: { ip: '127.0.0.6' }, // doc 2 with missing host.name field
	},
	];
	const secondExecutionDocuments = [
	{
	host: { name: 'host-a', ip: '127.0.0.10' },
	id,
	'@timestamp': secondTimestamp,
	},
	{
	host: { ip: '127.0.0.11' }, // doc 3 with missing host.name field
	id,
	'@timestamp': secondTimestamp,
	},
	];

	await indexListOfDocuments([
	...historicalDocuments,
	...firstExecutionDocuments,
	...secondExecutionDocuments,
	]);

	const rule: NewTermsRuleCreateProps = {
	...getCreateNewTermsRulesSchemaMock('rule-1', true),
	new_terms_fields: ['host.ip'],
	query: `id: "${id}"`,
	index: ['ecs_compliant'],
	history_window_start: historicalWindowStart,
	alert_suppression: {
	group_by: ['host.name'],
	duration: {
	value: 300,
	unit: 'm',
	},
	missing_fields_strategy: 'doNotSuppress',
	},
	from: 'now-35m',
	interval: '30m',
	};

	const { previewId } = await previewRule({
	supertest,
	rule,
	timeframeEnd: new Date('2020-10-28T06:30:00.000Z'),
	invocationCount: 2,
	});
	const previewAlerts = await getPreviewAlerts({
	es,
	previewId,
	sort: ['host.name', ALERT_ORIGINAL_TIME],
	});
	expect(previewAlerts.length).toEqual(4);
	expect(previewAlerts[0]._source).toEqual({
	...previewAlerts[0]._source,
	[ALERT_SUPPRESSION_TERMS]: [
	{
	field: 'host.name',
	value: ['host-a'],
	},
	],
	[ALERT_ORIGINAL_TIME]: firstTimestamp,
	[ALERT_SUPPRESSION_START]: firstTimestamp,
	[ALERT_SUPPRESSION_END]: secondTimestamp,
	[ALERT_SUPPRESSION_DOCS_COUNT]: 2,
	});

	// rest of alerts are not suppressed and do not have suppress properties
	previewAlerts.slice(1).forEach((previewAlert) => {
	const source = previewAlert._source;
	expect(source).toHaveProperty('id', id);
	expect(source).not.toHaveProperty(ALERT_SUPPRESSION_DOCS_COUNT);
	expect(source).not.toHaveProperty(ALERT_SUPPRESSION_END);
	expect(source).not.toHaveProperty(ALERT_SUPPRESSION_TERMS);
	expect(source).not.toHaveProperty(ALERT_SUPPRESSION_DOCS_COUNT);
	});
	});

[Security Solution][Detection Engine] adds alert suppression for New Terms rule type #178294

[Security Solution][Detection Engine] adds alert suppression for New Terms rule type #178294

Conversation

vitaliidm commented Mar 8, 2024 • edited Loading

Summary

UI

Checklist

elasticmachine commented Apr 2, 2024

vitaliidm commented Apr 2, 2024 • edited by rylnd Loading

Files by Code Owner

elastic/obs-ux-management-team

elastic/response-ops

elastic/security-detection-engine

elastic/security-detection-rule-management

elastic/security-detections-response

elastic/security-engineering-productivity

elastic/security-solution

pmuellr left a comment

Choose a reason for hiding this comment

MadameSheema left a comment

Choose a reason for hiding this comment

maximpn left a comment

Choose a reason for hiding this comment

rylnd left a comment

Choose a reason for hiding this comment

rylnd Apr 5, 2024

Choose a reason for hiding this comment

vitaliidm Apr 8, 2024

Choose a reason for hiding this comment

rylnd Apr 9, 2024

Choose a reason for hiding this comment

rylnd Apr 5, 2024

Choose a reason for hiding this comment

vitaliidm Apr 8, 2024

Choose a reason for hiding this comment

vitaliidm commented Apr 8, 2024

kibana-ci commented Apr 8, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

References to deprecated APIs

History

vitaliidm commented Mar 8, 2024 •

edited

Loading

vitaliidm commented Apr 2, 2024 •

edited by rylnd

Loading