Allow for Filtering the Log Context View

[weltenwort]

Note: This is a summary of the exploration and evaluation that is documented in weltenwort/kibana-plan-context-filter. It expands the work started in #9198 to fulfill a broader set of needs such as those described in the comments if #275.

Problem Statement

Centralized logging with Elasticsearch and Kibana is particularly valuable for large amounts of log data from numerous sources. For large amounts of log entries however, the current context view as implemented in #9198 loses most of its usefulness, because log entries not related to the anchor entry swamp the relevant entries in the context.

Discussion

The concept of the context view is to loosen the potentially very narrow filters used to find a particular event in the Discover view to allow for a broader inspection of the "relevant" events in the system immediately leading up to (or following) the event. This misleadingly suggests that the latter set of filters is a subset of the former, which might not even be the case in most circumstances. Another, maybe less misleading formulation would be that we want to provide a way for the user to tell Kibana what makes up the "relevant context" for the event so Kibana. So far the context view only uses adjacency in time as a criterion.

To summarize some observations about the two sets of filters:

• The set of filters to find the anchor document and the set of filters defining the context might be disjunct. (See Use Case 1 as an example.)
• In addition to adjacency in time, the context might be defined by more than one criterion (e.g. server + process name).
• Some problems might require investigation from different angles (i.e. different definitions of "context").

Abstract Formulation

It should be possible to limit the entries shown in the context view to those "relevant" to the event the user is inspecting. This translates to the ability to apply filters to the context view that are different from the filters used to identify the original log entry.

Concrete Solution Proposal

• A: The user uses the filters and query bar to find relevant events in the Discover view.
• B: The user expands a row and clicks the link that switches to the Context view.
• C: The context view is unfiltered, but contains disabled copies of the filters previously defined in the Discover view.
• D: The user re-enables some of the previously defined filters to narrow down the context. The view refreshes automatically.
• E: The user adds more term filters via the icons in the document details to narrow down the context. The view refreshes automatically.
• F: The filter bar contains the previous filters as well as the newly added filters, which can then be manipulated as usual.

[LucaWintergerst]

I do like solution 4 as it gives the user a possibility to use the feature without having to change their data.
For example if I want to "tail" a log file after I see a event in discover, I could configure the context filters to be "hostname" and "path" to filter it down to that one file.

While this makes it very convenient it requires that first step of configuring a filter set.
Perhaps adding a second button would help to work around this. One which switches without filtering and one where the configured filters are applied (or a drop down to choose one). The one where the filters are applied could only be clickable when a user configured a filter set.

Workflow 3 can be a little confusing if we mix it with saved searches, but I can also see the benefits of it, since it adds a lot of possibilities. I would still say to keep it simple and have dedicated filter definitions on the index pattern.

In any case modifying the filters after switching should also be supported (candidate #3) so I can filter down if I don't have a filter set or if I want to modify my filters.

[weltenwort]

@LucaWintergerst, thank you for that feedback!

I would be interested to also hear your opinion regarding the choice between variations D1/D2 or D4/D5 of solution candidate 4. D1 and D4 propose to store the filter sets per index pattern while D2 and D5 posit the saved search as the place to define and store the filter sets.

From your experience, how heavy would the following argument weigh for you?

CON: The filter set definitions can not be re-used across saved searches.

[LucaWintergerst]

My first thought was that using saved searches makes it confusing, unless we can filter by type. E.g. there will be regular saved searches and filter sets.

Something that would be nice to have would be filter sets across index patterns.
A filter set for "hostname + path" is obviously something that I will need for most index patterns sending log data. I would not want to add this 10 times. Wether it will be stored as part of the index pattern or as a saved search.
I understand that that's difficult and error prone.

If I just had to decide between index pattern and saved search, I would go with the index pattern. Defining a filter set is much different from defining a filter as part of a saved search. The user would probably get confused if it is part of the same

Does that make sense?

[LucaWintergerst]

The con you listed for D1
CON: The filter set definitions can not be re-used across index patterns.
does also apply to the searches (D2 D3) in my opinion. A search is always bound to an index pattern as well

[weltenwort]

You could "re-use/copy" a filter set definition from one saved search to the other by loading the search, changing the index pattern and re-saving under a new name. But that feels more like a workaround too. Index patterns really seem to be the best location so far.

Thanks again for the input!

[LucaWintergerst]

You are right, changing index patterns does work for searches. I was not aware of this until now.

I suppose that top level filter sets (for all patterns) require too many changes, as it would be easier to just tie them to either search or index patterns?

[weltenwort]

Introducing a new top-level concept could be quite an effort, yes. But more importantly, it could be pretty difficult to find a user workflow that is easy to understand and perform. More freedom often means a more complex UI with more potential for misunderstanding and confusion (e.g. applying the filters to index patterns that don't have the required fields). Restricting the scope allows for more guidance and sanity checking by the application.