[Lens] Introduce the concept of "related fields" to guide end-users

[wylieconlon]

This issue is not about implementing a specific proposal, but describing a concept that would improve the use of Lens for many types of data stored in Elasticsearch

Key concept: Fields can be related to each other for logical reasons, such as an ID field that is required for use by sub-metrics. This relationship is understood by the administrator who sets up the data, but not necessarily by a Lens user who wants to quickly create visualizations.

To give an example of the difference between what an administrator knows and what the end-user knows, imagine that an administrator has installed Metricbeat to measure metrics for Elasticsearch. Metricbeat has inherent structure to how it organizes its data into "modules" and metric sets, which is reflected in the naming of the fields. As an end-user you can guess that fields with similar names are related, but this is not as easy to use as Lens should be. Even worse, some datasets have "gotchas" that are not easily guessed at.

There are a few common types of relations that would benefit users:

• Wildcard group: Creates a grouping of "related fields" with a regular expression
• Bucketing on a field that will include/exclude other fields from the results
  • For example, when bucketing by Index Name you automatically exclude Shard-related fields
• Related split by: Another field to highlight to "break down" this one, such as the ID or Name field
• Recommended pairing: Another field that often appears together in the visualization, for example "download" could pair with "upload"

Where to implement this?

The goal of "related fields" is to let administrators manage the relations so that the end-users get a good experience. An advanced enough end-user might also be able to configure related fields.

There are two main candidates for where to implement this:

• Extend the functionality of index patterns
• Introduce a new saved object type for Lens which depends on index patterns
• Use the _meta field in index mappings to store relation

No matter which of these is chosen, I would expect that Beats and the Ingest manager would take an active role in defining the relationships.

Alternatives to this proposal

Instead of defining the relationships manually, it might be possible to learn over time about how fields are used by tracking user behavior and building a recommendation system, but I think the accuracy would be higher on a manually-defined relationship than a learned one.

cc @mattkime @exekias

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[ghudgins]

will track this idea as a project on overall planning boards and in #17087