Change in global filters within the URL breaks portability of ML jobs using custom URL links to Kibana dashboards.

[robcowart]

Kibana version: 6.0.0-beta2

Elasticsearch version: 6.0.0-beta2

Server OS version: MacOS

Browser version: Chrome

Original install method (e.g. download page, yum, from source, etc.): tar from download page

Description of the problem including expected versus actual behavior:
In version 5.x the global filters portion of the URL accepted an index pattern name. For example:

kibana#/dashboard/ca480720-2fdf-11e7-9d02-3f49bde5c1d5?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'netflow-*', ...

In version 6.0.0-beta2 it appears that an index pattern ID is used. For example:

kibana#/dashboard/5feb9f70-9355-11e7-b655-4b7143aa497d?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'17bced90-934a-11e7-b655-4b7143aa497d', ...

Interestingly if an index pattern name is used in 6.0.0-beta2 the data will be filtered appropriately. However the filters are not visible in UI and cannot be changed in any way. NOTE: This was not an issue in 5.x (at least 5.4 and later).

If using the ID of the index pattern the data is still filtered properly and the filters appear in the UI.

Here is where this becomes a usability problem. In ML there is the ability to define a custom URL. Variables provided by the detected anomaly can be injected into the URL to allow for a launch in context to the data which produced the anomaly. An example of such a custom URL is...

kibana#/dashboard/ca480720-2fdf-11e7-9d02-3f49bde5c1d5?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'netflow-*',key:netflow.server_addr,negate:!f,type:phrase,value:'$netflow.server_addr$'),query:(match:(netflow.server_addr:(query:'$netflow.server_addr$',type:phrase))))),time:(from:'$earliest$',mode:absolute,to:'$latest$'))

This example works in 5.x. Notice the parameter index:'netflow-*'

To make this work in 6.x the index parameter would have to be changed to the ID of the index pattern. For example: index:'17bced90-934a-11e7-b655-4b7143aa497d' However, once this change is made, this ML job can no longer be easily imported into another cluster as the ID of the index pattern will be different, even though the index pattern name is the same.

As it is at the moment users will have a choice. They can either have portable ML jobs (e.g. between DEV, QA and PROD environments) or they can have visible filters in the UI.

In 5.x they could have both. 6.x should restore this behaviour to that of 5.x.

Steps to reproduce:

1. Create a dashboard and set a filter.
2. Edit the URL manually replacing the index pattern ID for the filter with the index pattern name.
3. Refresh the browser. The data will still be filtered, but the filters are no longer displayed.

[Bargs]

The invisible filter problem was reported in #7767. I'm working on a fix for it in #13916. The problem is that the index pattern ID is invalid. Instead of making filters with invalid indices invisible (which was an unintended bug) we'll need to show the filter, indicate the issue to the user, and allow them to fix it. We can't just use the index pattern title because multiple index patterns might have the same title and we wouldn't know which to load. This solution is better than having invisible filters, but I imagine it won't be entirely satisfactory for your use case.

You could alternatively try using our new query language in 6.0. Instead of injecting variables into a filter, inject them into the query string itself. Kuery doesn't require you to specify an index, so you won't run into this problem and the ML jobs will be fully portable.

[robcowart]

When you say "index title" do you mean this "netflow-*"? If so, I am not sure why this is invalid. Kibana won't let you create to index patterns with the same name/pattern.

Regarding Kuery... not an option unless the 6.0.0 upgrade process will automatically rewrite the URLs of existing ML jobs.

[Bargs]

Yep, that's the title I meant. Unique titles are a limitation now, but might not be in the future.

You could insert variables into the lucene query string prior to 6.0.

That said, I'm working on a fix that might allow filters with invalid indices to continue to work. Please follow #7767 and #13916 for any updates on that front.