[Security Solution] Detection Rules' max_signals parameter is not respected beyond 1000

[rylnd]

Describe the bug:

Kibana/Elasticsearch Stack version:
8.8

Steps to reproduce:

1. Create a rule with max_signals set to 10,000
2. Execute the rule such that it would create >1000 alerts
3. Observe the rule fail

Current behavior:
Up to 1000 alerts are generated, and the rule has an execution error. max_signals is not respected, and the default alerting value of 1000 is used to cap/validate the number of signals.

Expected behavior:
Up to 10,000 alerts are generated. If that number is exceeded, a warning is created on the rule.

Screenshots (if relevant):

Related Issues

While not confirmed, I believe that the work done for #153611 introduced this behavior.

Slak conversation

https://elastic.slack.com/archives/CN7AD3605/p1683741078780019

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[e40pud]

We added docs about this behaviour elastic/security-docs#3293

@yctercero I guess we can consider options to fix this in 8.9?

[yctercero]

It'd be great to get to this one 8.9 if possible. I'm not sure how many users are setting their max signals that high - maybe we have the telemetry @peluja1012 ? But would still be great to understand.

[yctercero]

@banderror I think your team is dealing with this in the current prebuilt rule customization work - is that accurate?

[banderror]

@yctercero Yes and no.

We worked on #173593 and fixed #164234. We also noticed the issue described in this ticket and had some discussion with PM and TRaDE. Summary of the discussion can be found here. Immediate action items we've already addressed were:

• We agreed that it would be worth documenting that those rules that have max_signals > xpack.alerting.rules.run.alerts.max will be generating at most xpack.alerting.rules.run.alerts.max alerts per execution (1000 by default).
  • @joepeeples documented this in the public docs (PR).
  • @Mikaayenson updated setup guides of those 18 rules with that information (PR).

No more work related to this is planned within the rule customization epic.

But I agree that it's an issue and non-friendly and obscure UX. I'm just unsure that it should be handled as described in the expected behavior section in this ticket. The xpack.alerting.rules.run.alerts.max setting seems to be a hard cap by design, because it's a global circuit breaker. We could handle this issue by changing the UX of installing and enabling rules. If we do it like that, the issue will be on the Rule Management side.

cc @approksiu

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[yctercero]

Thanks for the quick response @banderror !

We could handle this issue by changing the UX of installing and enabling rules. If we do it like that, the issue will be on the Rule Management side.

By this, do you mean just updating the UI to add validation so users can't set anything above xpack.alerting.rules.run.alerts.max and updating the prebuilt rules to be below that threshold? I know it could be a moving target since it's a setting but perhaps setting the prebuilt rules to the default xpack.alerting.rules.run.alerts.max is what I imagine?

[banderror]

By this, do you mean just updating the UI to add validation so users can't set anything above xpack.alerting.rules.run.alerts.max and updating the prebuilt rules to be below that threshold?

@yctercero No, we already have validation and it's a warning in this case (user can still create or update the rule). Also, updating the 18 prebuilt rules that have max_signals == 10000 is not an option.

The idea is that the app should give a heads up and explain to the user that if they want to use certain prebuilt rules, they will need/want to increase the global setting to allow these rules to promote as many external alerts as possible. I think how users increase this global setting is a separate question. Maybe, we can make it easier for users via Kibana Advanced Settings.

But it's all speculation from my side. Let's wait for input from @approksiu when she has a chance to think about the UX for this feature.