[Logs UI] Add dataset-specific categorization warnings

[weltenwort]

Summary

This adds dataset-specific categorization warnings for the categorization module. The warnings are displayed in call-outs on the relevant tabs as well as the job setup screens if a prior job with warnings exists. To that end this also changes the categorization job configuration to enable the partitioned categorization mode.

closes #60392

Previews

Implementation notes

This also replaces the locally defined setup fly-out on the categories tab with the shared fly-out. To avoid leaking the log rate analysis module into that tab the fly-out can now be configured with an allow-list of modules to display.

@elastic/kibana-operations and @elastic/ml-ui were tagged as codeowners on this PR because I fixed one line in the storybook configuration and updated one of the ml module definitions created for the Logs UI.

Testing

• This PR includes several storybooks to render the warning call-out and setup view in various states. Run yarn storybook infra to build it.
• The categorization job configuration changes will only take effect for new jobs. A call-out should be displayed to the user.
• Getting the ML jobs into a "warn" state is tricky. One way would be to ingest mostly identical log message so only one category is created. At least 100 documents are required to trigger any warning.
• flog is a useful tool to generate log entries. They can be ingested using filebeat with a processor config such as

  processors:
    - dissect:
        tokenizer: "<%{syslog_priority}>%{syslog_version} %{@timestamp} %{message}"
        field: "message"
        target_prefix: ""
        overwrite_keys: true
    - add_fields:
        target: "event"
        fields:
          dataset: "static.test"

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[spalger]

Operations LGTM, just fixes the location of the infra storybook

[afgomez]

Thanks for the adding the component to storybook! It was useful for the first pass 👍

I have a comment regarding the layout. When the description of an issue in a dataset is too long, it wraps to the next line. If a dataset has several issues it looks a bit chaotic.

Would it make sense to group the warnings per dataset? Something like:

**first.dataset**
• The analysis couldn't extract more than a single category from the log messages.
• 95% of the categories only rarely have messages assigned to them.
• The ratio of categories per analyzed document is very high with 0.7.

**second.dataset**
• None of the extracted categories frequently have messages assigned to them.
• 70% of the categories won't have new messages assigned to them because they are overshadowed by less specific categories.

[afgomez]

I did a first pass on the code. I left some small comments.

[afgomez]

Was there a reason to keep this alias? Otherwise I think we can safely remove it.

[weltenwort]

Well, the thinking was that the CategoryQualityWarning is just a special case. I wanted to make that clear in the consuming code that it shouldn't rely too strictly on the fact that no other warnings have been implemented.

Other warnings could be that buckets have been skipped due to delay, which can happen in any anomaly detection job.

[afgomez]

Code looks alright! I found a crash on an API call (see comment below).

Since we are creating the types ourselves I wonder if there are other things we missed. Maybe the ML team has shared types that we can reuse.

[afgomez]

I added patches for some of the comments I had.

[afgomez]

I addressed the changes I requested... so I approve my own changes!

[weltenwort]

@afgomez thanks for the review and the grouping suggestions. Unfortunately the margin-top pushes the content outside of the accordion, so I'll have to solve that differently:

[weltenwort]

@afgomez I modified the DOM structure to use a definition list, which looks almost the same as your solution. But it doesn't have the margin problem and doesn't skip heading levels for accessibility.

I updated the preview in the description. Does this look good to you?

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 87ca95e

Metrics [docs]

@kbn/optimizer bundle module count

id	value	diff	baseline
infra	1193	+69	1124

async chunks size

id	value	diff	baseline
infra	4.2MB	+12.3KB	4.2MB

page load bundle size

id	value	diff	baseline
infra	277.4KB	+25.0B	277.3KB

distributable file count

id	value	diff	baseline
default	45883	+5	45878

History

• 💚 Build #76537 succeeded 36bdcbc
• 💚 Build #76533 succeeded c28a749
• 💔 Build #74439 failed 404227e
• 💚 Build #71838 succeeded d11d530
• 💔 Build #71580 failed 1a09822

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream