[Security Solution] Implement per-field text-based diffs for the prebuilt rule upgrade flyout

[nikitaindik]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174167
Related to: #169160

Summary

The initial plan was to build a new tab in the prebuilt rule upgrade flyout where we'd show per-field diffs for rule fields using custom diff components. These custom components were designed within #166159. This ticket was originally for implementing these custom components.

To move faster and implement this tab as soon as possible, the team has decided to first implement it using text-based diffs based on the same mechanism used for showing JSON diffs (#169160).

Acceptance criteria

• The tab with per-field diffs is hidden behind a new feature flag. When the flag is off, the tab does not appear in the flyout. The tab should work regardless of the value of jsonPrebuiltRulesDiffingEnabled.
• Per-field diffs are read-only components. We don't need to let the user "merge" differences using these components.
• Diffs for complex fields are rendered as JSON diffs using the same component used for rendering the JSON diff for the whole rule. This means this component should be abstracted away and should accept unknown values in props instead of RuleResponse.
• Diffs for related fields are grouped or rendered close to each other. For example:
  • Index patterns + Data view id
  • Custom query + Filters + Language + Saved query id
• The tab uses the response from the upgrade/_review API endpoint and doesn't need any other API calls to render itself.
• The tab renders itself under 150ms.

Release progress

• Initial implementation is done but the feature is hidden behind a feature flag. (PR)
• Feature is covered with a test plan and automated tests written according to it. (test plan PR, tests PR)
• Feature tour is implemented. (PR)
• Feature is fully implemented and considered by the development team as ready to be released.
• Acceptance testing is done and the feature is approved by @approksiu and @ARWNightingale.
• Exploratory testing is done and the feature is approved by @vgomez-el.
• Documentation is written for ESS and Serverless by @joepeeples. Two docs PRs are approved and ready to be merged. (ticket)
• Feature flag is turned on by default. (PR)
• Feature is released in Serverless.

Planned release date in Serverless: March 4th.
Planned release date in ESS: March 19th (v8.13.0).

Designs

Latest Figma Designs

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

@dplumlee I updated the ticket description according to our recent discussions. cc @approksiu @nikitaindik @jpdjere

[dplumlee]

As a follow-up to the discussion held in the meeting yesterday, I'm listing out the field groups we are displaying in the upgrade flyout so we can determine how to order them. Currently, there's no discernible order to the fields, but we talked about either matching the order to that of an existing pattern (e.g. the field order of the rule details flyout or the rule creation form) or an order of "importance" (e.g. query fields more important, description fields less important). @approksiu @ARWNightingale

"Common" Fields

• name
• tags
• description
• severity
• severity_mapping
• risk_score
• risk_score_mapping
• references
• false_positives
• threat
• note
• setup
• related_integrations
• required_fields
• author
• license
• rule_schedule
• actions
• throttle
• exceptions_list
• max_signals
• type

"Per rule type" fields

(these field groupings will all be displayed in the same component for context)

• kql_query
  • type
  • query
  • language
  • filters
• eql_query
  • query
  • language
  • filters
• event_category_override
• timestamp_field
• tiebreaker_field
• esql_query
  • query
  • language
• threat_query
  • type
  • query
  • language
  • filters
• threshold
  • field
  • value
  • cardinality
• machine_learning_job_id
• anomaly_threshold
• new_terms_fields
• history_window_start
• data_source
  • type
  • index_patterns OR data_view_id

[approksiu]

Reviewing.

[dplumlee]

@approksiu #176767 here is the PR for the tour

[dplumlee]

@vgomez-el we're hoping to get the remaining release progress steps checked off soon (ideally by end-of-week), do you have everything you need for exploratory testing on this?

[vgomez-el]

@dplumlee Just to clarify, will the feature be included in the next 8.13 BC2 planned for Feb 22nd? Or should I test the feature on the feature branch?

[banderror]

@vgomez-el The feature is already available in 8.13 but hidden behind a feature flag. We can enable the feature flag by default today to make it enabled in BC2 w/o any extra configuration. However, we'd prefer testing on the 8.13 branch ASAP to get feedback faster. We only have 3 days left before the end of the week to complete all the work from the release progress checklist (see the description), and in case of any bugs found we'll need some time to fix them. Sorry for the short notice, we should have given a heads up last week.

The flag can be set in the kibana config this way:

xpack.securitySolution.enableExperimental: [
  'perFieldPrebuiltRulesDiffingEnabled'
]

[vgomez-el]

@banderror @dplumlee I have exploratory tested the feature activating the feature flag in a cloud environment after upgrading a 8.9.2 instance to 8.13 so I have several prebuilt rules to work with.
I have tested on a first round it using an admin user and a read-only user and everything worked fine. I have already submitted my approval for this task.
thank you!

[banderror]

@ARWNightingale @dplumlee How's it going with acceptance testing?

[ARWNightingale]

@banderror all looks good and everything as expected.

[banderror]

The feature went live today 🎉