[index patterns] Index pattern “pattern list” support

[stephmilovic]

The Ask

In order to take advantage of runtime fields, the Security Solutions team needs to adopt Kibana Index Patterns. In order to adopt Kibana Index Patterns, the Security Solutions team needs to define a pattern list that will work whether or not the user has the matching indices for each pattern on the list.

The Problem

Currently, a user can create an index pattern with a list of index patterns, ie: { title: "filebeat-*,auditbeat-*" }. Unfortunately, when a user creates a pattern list with a pattern that does not match data { title: "fake-*,filebeat-*,auditbeat-* "}, the pattern blows up:

The Solution

We are not validating that each pattern on the list matches indices when fetching fields, and the getFieldsForWildcard function is not prepared to handle that. We need to hand getFieldsForWildcard a list of validated, "active" patterns only.

• Restructure index-pattern saved object to accommodate patternList
  • title remains as a label for the index pattern
  • title will be migrated to patternList from a string to a string array, separated by commas.
    • ex: { title: "filebeat-*,auditbeat-*" } becomes { title: "filebeat-*,auditbeat-*", patternList: ["filebeat-*","auditbeat-*"] }
  • going forward, title will be a descriptive field
    • ex: { title: "Security Solution Indices", patternList: ["filebeat-*","auditbeat-*"] }
• at runtime, detects index patterns in the patternList that do not match data and returns patternListActive
• Security Solution index pattern will be editable and deletable. The pattern will have a unique ID and if it is not present when the Security Solution app is loaded, it will be created.

Usage in other apps that (might) need adjustment:

• APM
  • Update title to patternList in createStaticIndexPattern ([APM] Index pattern: Rename title to patternList #89985)
• Discover:
  • Sidebar index pattern selector UI
• Lens:
  • Index pattern selector UI
• Fleet:
  • createIndexPattern method updates to use patternList
• TSVB:
  • Reports used index patterns to dashboard
• Timelion:
  • Autocompletion of field types
  • Autocompletion of index patterns names
  • Background search (for scripted fields)
• Vega:
  • Reports used index pattern to dashboard
  • Potentially more logic when merging filters into the spec?
• Graph:
  • Uses index patterns for the user to select (and then custom query logic)
• Dev Tools:
  • Is autocompleting on index patterns
• Security Solution:
  • Disable sourcerer import of Kibana Index Pattern. Problem will fix itself when this issue is completed: https://github.com/elastic/security-team/issues/772
• Alerting:
  • index threshold alert can use index patterns in the typeahead when selecting an index to query
  • index document action uses index patterns (via typeahead) when selecting an index (or index pattern) to write documents to
• Reporting:
  • Index patterns api is used by CSV generation <- not sure if change is actually required
• Ingest:
  • creates index patterns, method needs updates to use patternList
• ML:
  • Index data visualizer
  • File data visualizer, index pattern creation
  • Anomaly detection job wizards. Displaying of index patterns name and adding the indices to the datafeed.
  • Data frame analytics job wizards
    • Reading index patterns
    • Creating index patterns
  • Clone anomaly detection job
  • Clone data frame analytics job
  • Anomaly Explorer anomaly charts
  • Single Metric Viewer metric data plots
  • Data recognizer modules (default index pattern configuration)
• Transforms
  • Create Transform wizard
  • Clone Transform

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[stephmilovic]

Here are some screen recordings of the index pattern management UI changes

Disclaimer: WIP

Please provide feedback in the comments :)

• user cannot create a duplicate title pattern, but can create a pattern with the same pattern list
  
• user can select a combination of active patterns (matches indices) and inactive patterns (does not match any indices), but needs at least one active pattern to create the Kibana Index Pattern
  
• getFieldsForWildcard now joins fields from each pattern together
  
• when a KIP is called, an activePatternList is generated at runtime. only active patterns are queried.
  

[stephmilovic]

Discover UI changes

Disclaimer: WIP

Please provide feedback in the comments :)

1. I didn't add an info icon to the button label, the ToolTip is on the whole button. do you think i need an info icon?
2. What do you think about the EuiBadge for the pattern list in the ToolTip? this is how I've been displaying the patternList/patternListActive concept. i had a ToolTip to explain the 2 states of the badges, but obviously doesn't work with ToolTip+ToolTip
3. In the dropdown, I'm doing patternList.join(', ') . 2 part question here. Do you think I should do patternListActive.join(', ') and not even show the non-active patterns? Should the join include a space or not? (auditbeat-*, filebeat-* or auditbeat-*,filebeat-* )

[andreadelrio]

@stephmilovic I'd like to see this in the browser but while we figure out local setup here are some preliminary comments.

Discover UI changes

Disclaimer: WIP

Please provide feedback in the comments :)

1. I didn't add an info icon to the button label, the ToolTip is on the whole button. do you think i need an info icon?

A tooltip on the whole button might be a bit intrusive. I suggest showing the tooltip only when hovering an info icon.

2. What do you think about the EuiBadge for the pattern list in the ToolTip? this is how I've been displaying the patternList/patternListActive concept. i had a ToolTip to explain the 2 states of the badges, but obviously doesn't work with ToolTip+ToolTip

I'd suggest sticking to plain text inside tooltips. EuiBadge feels a bit overkill for this use case.

3. In the dropdown, I'm doing patternList.join(', ') . 2 part question here. Do you think I should do patternListActive.join(', ') and not even show the non-active patterns? Should the join include a space or not? (auditbeat-*, filebeat-* or auditbeat-*,filebeat-* )

3.1 If you're showing the info of non-active patterns elsewhere I'd say keep it out of this tooltip that way we keep the content short. 3.2 I would include a space.

[jgowdyelastic]

ML uses the index pattern title in a lot of places. We will have to modify these to use a joined patternList when creating Anomaly Detector and Data Frame Analytics jobs and when using the index data visualizer.
We also display this title to the user in the new job wizards, so they know which indices they are using. If we move to title being descriptive text, we may have to show this as well as the the joined list of indices.

When cloning a job we have to guess at which index pattern was originally used, we do this by joining the datafeed indices and matching this against index pattern titles. This could change to just matching the datafeed's indices against the patternList or we could investigate keeping a reference of the original index pattern in our recently added job saved objects.
I believe we would still need to fall back to guessing the index pattern if this saved object reference has been lost or the job has been migrated from a previous version or the job has been created directly in ES.

[stephmilovic]

Closing in favor of: https://github.com/elastic/security-team/issues/772