Explosion of mapping fields exceeds default 1k limit on upgrade

[tylersmalley]

We recently had a failure on CI due to our total fields exceeded 1k during an upgrade.

Error: Limit of total fields [1000] in index [.kibana_2] has been exceeded

In this situation, the security solution is defining a mapping for the Kibana index from a few months ago. During the functional tests, the migrations are ran on this index due to it being out of date. Migrations will continue to persist previous mappings into the future index to ensure we can copy data from plugins which are unavailable. There was recently a rename for the maps-telemetry object type which plays a part in this (#69816).

@aaronjcaldwell, we might want to consider reverting #69816 for the reason mentioned above until we have a solution.

Additionally, the maps-telemetry type defines two properties with dynamic: true. We should prevent this, as it can greatly effect the number of fields which we have. An alternative option if the plugin is outside of OSS, would be to use the flattened type which might be more fitting.

I am marking this a blocker for 7.9 as it seems likely users who are upgrading would encounter this as well.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[kindsun]

Sorry for the trouble! This ended up being slightly more than a revert so I have a PR up (#69871) that should address any issues from our end. In the meantime, we'll see if we need to tackle this a different way.

…

On Wed, Jun 24, 2020 at 6:10 PM Elastic Machine ***@***.***> wrote: Pinging @elastic/kibana-platform <https://github.com/orgs/elastic/teams/kibana-platform> (Team:Platform) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#69869 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANGDUEIY2Y3YVNBZ2R6YBLRYKIYJANCNFSM4OHQZRLA> .

[pgayvallet]

Migrations will continue to persist previous mappings into the future index to ensure we can copy data from plugins which are unavailable

Yea. This is far from being something we can easily change atm tbh.

@rudolf correct me if I'm wrong, but this is not even something that's scoped in the improved SO migration, right?

[rudolf]

@pgayvallet yes, we have an issue for this but not yet a good solution #67086. It will make sense to add this to the migration algorithm, we should probably set the mappings of any unknown saved object type to enabled: false to clean it up (this will require a re-index).

@aaronjcaldwell Renaming a type will add new fields for the new type name, but leave the mappings of the old type in place, so it effectively doubles the field count for your type. However, that in itself is unlikely to cause the mapping explosion, it seems to me like the real problem is the dynamic: true mappings.

If we don't know the shape of the data we're storing, we also can't query it, so I don't think there's a good reason to index this data. We should probably set these to enabled: false https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html so that the data is only stored not indexed. Does Kibana ever query any of this telemetry data directly? If we just store and forward it to the telemetry cluster we could set enabled: false for the entire maps / maps-telemetry saved object type.

If we do wish to rename this type but still clean up the previous type name's mappings we should do the following:

1. Keep the maps-telemetry type, but set it's mappings to enabled: false so that we're no longer indexing any of that data (with a comment explaining that this type is only registered to cleanup).
2. Define a new maps type. Note that any maps-telemetry saved objects will be "lost" when Kibana gets upgraded so we'll probably loose all the data that hasn't been uploaded yet. This might be fine for telemetry data, but it's one of the reasons renaming types aren't really supported.

[kindsun]

@rudolf Ok, good to know. It also may not be necessary in this case. Per @afharo's recommendations I've reverted the SO name from maps back to maps-telemetry in #69871 and rather than remove the dynamic fields or change their enablement, revised them to not be dynamic. Feel free to comment on this PR if you think this covers us or if there's anything further I need to do!

Does Kibana ever query any of this telemetry data directly?

I don't believe so. To the best of my knowledge indexing of this data was disabled in cloud