Cannot import saved objects from 6.8.1 to 7.3.0 if the object has filter aggregations

[bhavyarm]

Kibana version: 7.3.0 BC4

Elasticsearch version: 7.3.0 BC4

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: If user tries to import metric viz with filters aggregation from 6.8.1 to 7.2.0 - Kibana fails to do so and logs:

Exception @ migrations730 while trying to migrate query filters! TypeError: Cannot read property 'query' of undefined

Steps to reproduce:

1. Load sample data from load sample data kibana tutorial for logstash - create logstash-* index pattern https://www.elastic.co/guide/en/kibana/7.3/tutorial-build-dashboard.html#tutorial-load-sample-data
2. Import the attached json and associate the saved objects - machine_os horizontal_bard_chart response_code with logstash index pattern
3. Open the dashboard all_the_viz
4. Metric viz fails to display and logs [esaggs] > Cannot read property 'doc_count' of undefined on screen

Screenshots (if relevant):

Errors in browser console (if relevant):

commons.bundle.js:3 Uncaught (in promise) TypeError: Cannot read property 'type' of undefined
    at _callee$ (commons.bundle.js:3)
    at tryCatch (vendors.bundle.dll.js:541)
    at Generator.invoke [as _invoke] (vendors.bundle.dll.js:541)
    at Generator.prototype.<computed> [as next] (vendors.bundle.dll.js:541)
    at asyncGeneratorStep (commons.bundle.js:3)
    at _next (commons.bundle.js:3)
    at commons.bundle.js:3
    at new Promise (<anonymous>)
    at commons.bundle.js:3
    at mapPhrases (commons.bundle.js:3)

Provide logs and/or server output (if relevant):

Kibana logs:

log   [12:16:58.112] [info][server][Kibana][http] http server running
  log   [12:22:29.247] [warning] Importing saved objects from a .json file has been deprecated
  log   [12:22:29.428] [warning][migrations] Exception @ migrations730 while trying to migrate query filters! TypeError: Cannot read property 'query' of undefined
  log   [12:22:29.432] [warning][migrations] Exception @ migrations730 while trying to migrate query filters! TypeError: Cannot read property 'query' of undefined

bubbles.json.txt

[elasticmachine]

Pinging @elastic/kibana-app

[bhavyarm]

Not fixed yet.

[bhavyarm]

cc @marius-dr

[bhavyarm]

Updated the headline as the rest of the saved objects are fine.

[stacey-gammon]

Hm, when I import bubbles.json into 7.3, it seems to work with no errors. I'm not using a snapshot though, I just did git checkout upstream/7.3

[bhavyarm]

@marius-dr can you please confirm the status of this bug either on BC6 or on today's snapshot please? I was on yesterday's and today's was still building. Cc @LeeDr

[stacey-gammon]

I checked on BC5 and still couldn't repro. :(

[stacey-gammon]

@marius-dr wasn't able to repro that original error either, so I think this can safely be closed. Maybe @bhavyarm you accidentally were running an older version?

[LeeDr]

I reproduced it on 7.3.0 BC6. A couple of things I tried;

1. If I take the request from the metric_filter visualization and paste it into the Dev Tools it works and I get back a good response. Here's that response;

{
  "took": 52,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 14005,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": {
        "{\"query_string\":{\"query\":\"machine.os.keyword:\\\"win 8\\\"\"}}": {
          "doc_count": 2838
        }
      }
    }
  },
  "status": 200
}

If I change the query from lucene to KQL, then it works;

And the response has changed to;

{
  "took": 24,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 14005,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": {
        "machine.os.keyword:\"win 8\"": {
          "doc_count": 2838
        }
      }
    }
  },
  "status": 200
}

If I change the filter back to lucene I get this error;

So it's strange that when I open the visualization and the filter is using lucence I initially get this error;

but after changing it to KQL and back to lucene I get the different error.

[stacey-gammon]

Should this be filed under a new issue? That error seems completely different from the error that showed up when importing the exported json from 6.8 to 7.3.

[LeeDr]

Looks like there were as many as 3 issues in the original description.

1. [esaggs] > Cannot read property 'doc_count' of undefined on screen (this is the same thing I just reproduced on the metric_filter visualization which uses a filter aggregation).
2. Exception @ migrations730 while trying to migrate query filters! TypeError: Cannot read property 'query' of undefined I did not reproduce this
3. commons.bundle.js:3 Uncaught (in promise) TypeError: Cannot read property 'type' of undefined I think this may indicate a problem on another visualization on the dashboard? I focused only on the metric_filter visualization and didn't see this.
4. [esaggs] > Cannot convert undefined or null to object I got this after I changed the query type on the filter agg from Lucene to KQL and back to Lucene. I think it's part of problem # 1. If I re-open the metric_filter visualization again I get the 'doc_count' of undefined.

[bhavyarm]

@LeeDr the same import works fine in 7.2.1

[bhavyarm]

This is the exported saved objects for 7.2.1 for the same objects in 7.3.0 bug
(I didn't select yes to Include related objects because that gives an error):
export_7.2.1.ndjson.txt

[bhavyarm]

@stacey-gammon @LeeDr just logged another bug: #42343 I don't think its related but just in case.

[elasticmachine]

Pinging @elastic/kibana-app-arch

[stacey-gammon]

tagging app arch team too because the bug may have been exposed from some of the filtering refactoring. filter.meta can be undefined. I don't think this is anything new from the migration step.

Might have something to do with this PR: https://github.com/elastic/kibana/pull/35544/files though nothing very obvious, though I do see this line filter.meta = filter.meta || {}; which should make that not undefined, but it's only after the mapping functions. I still don't see how that PR would have caused the issue, looks the same logically as the previous iteration, but maybe I'm just not seeing it right away.

[lukeelmers]

If I change the query from lucene to KQL, then it works;

If I change the filter back to lucene I get this error;

I was able to reproduce this on 7.3 but not master. On master if I toggle from lucene to KQL or vice-versa and hit apply, it will clear my filter entirely. In 7.3, it will clear the filter from the input, but I get "cannot convert undefined or null to object". Then if I save, I get that same error each time I re-open the vis. (But notably, not the doc_count issue which I think must specific to the dataset provided here).

So I think we should split this particular item out into a separate issue (item 4 from @LeeDr's list above), as it doesn't appear related to migrations and seems to just be a 7.3 problem. Haven't tested on earlier versions.

@lizozom Any thoughts on this or why it might be working on master? I think you've merged a few bugfixes related to filters recently.

[LeeDr]

I guess we need to also test if there are problems on 7.x (7.4.0).

[bhavyarm]

I will upload 7.x results shortly. Ran into a different problem.

[bhavyarm]

Same bug on 7.x @lukeelmers @LeeDr I am on the snapshot from Build #75 (Aug 2, 2019 5:41:00 AM)

[bhavyarm]

@LeeDr I couldn't reproduce this on 7.0.0/7.1.1

[lukeelmers]

Confirmed on 7.x that I can reproduce the issue based on the steps described above.

Spent awhile looking at this today and came up empty so far. I saw the same things @stacey-gammon noticed -- this originates from FilterManager, specifically map_phrases.js, but it does look like a default filter.meta is set in map_filter.js. However it is set after mapFn is called... which is where the error is coming from.

It seems this area of code was most affected by #35544 -- however the tags in that PR suggest it was merged for 7.2, which wouldn't explain why @bhavyarm is reporting this as working in 7.2 but not 7.3

@lizozom Any ideas on any of these items, since you currently have the most expertise on the FilterManager?

[bhavyarm]

@lukeelmers checked on 7.2.1 again and its definitely working. cc @LeeDr Thanks!

[elasticmachine]

Pinging @elastic/kibana-platform

[rudolf]

Ignore me, I was looking at the wrong ticket.

[zzzuzik]

We bumped our k8s cluster to 7.3 and was hit hard by this issue. All filters were set to Lucene, if we bulk edit exported virtualization to set language to Kuery, after importing them back GUI says KQL but all filters are gone (empty). We cannot downgrade since there are indices already created and ES refuses downgrade.
Any ideas how we can edit our exported visualizations to have both KQL and filters preserved? (more than 100 of them)

[lizozom]

@stacey-gammon @lukeelmers I debugged the issue, thinking this was a filter_manager issue, because the error is coming from filter manager (map_pharses).

However, the dashboard shouldn't have filters at all (it doesn't have them in 7.2). The problem happens when in 7.3, this dashboard decides it has a filter and then adds a misshaped one to $scope.filters. filter_state_manager picks in up, and calls setFilter and that's where the error in the console log is coming from.

I didn't figure out why the dashboard thought it had a filter though.

Regardless, we should finally TS the filter manager helper functions. :)

[stacey-gammon]

I think this issue is getting too confusing with all the different bugs going on. I split it out into the two ones that are left:

#43188 and
#43189