Migrations always run if there is an old index template

[chrisdavies]

Latest master, running against a shared cloud ES instance that has an old index template.

The pattern is adding this mapping: gis-map.properties.bounds.strategy: 'recursive', which no longer exists in the latest version of Kibana. As a result, each time Kibana restarts, it sees that the Kibana index has an old mapping and attempts to migrate.

I think the solution is to remove / disable .kibana index templates prior to running migration, but I'm uncertain what potential negative consequences this might have.

Repro steps

• Start Kibana in repl mode: yarn start --repl
• After Kibana is done booting up, paste in the following:

var callCluster = kbnServer.server.plugins.elasticsearch.getCluster('admin').callWithInternalUser;

callCluster('indices.putTemplate', {
  name: 'kibana_index_template',
  body: {
    index_patterns: '.kibana',
    mappings: {
      "doc": {
        "dynamic": "strict",
        "properties": {
          "gis-map": {
            "properties": {
              "foo": {
                "type": "text",
              },
            },
          },
        },
      },
    },
  }
});

• Delete the Kibana index (e.g. callCluster('indices.delete', { index: '.kibana_1' }) or whatever)
• Restart Kibana
• Note it migrates, restart again, and again, and it migrates each time...

NOTE: if you add "dynamic": "strict" to the index template mappings, the migration will fail (because it's adding fields to the mapping).

[elasticmachine]

Pinging @elastic/kibana-operations

[chrisdavies]

Seems like Tyler is addressing this specific instance with this PR: #28038

[chrisdavies]

Seems we may want to remove the putTemplate call from the saved objects service while we're looking into this: #23661

Edit: @spalger pointed out that this is a breaking change, and should be 7.x or 8.x if we can't get it into 7.x

[chrisdavies]

We discussed this at length yesterday, and came to a few different conclusions.

We should delete the Kibana index template prior to migrating, as we know it contains the old mappings, and that will cause unwanted interference with migrations.

But, that's only a partial fix. There still could be other index templates that target the Kibana index, and ES seems to have some magical mapping properites that will just reappear even though they weren't eplicitly stated at creation time: #28038

We need a less fragile way of determining if mapping changes should trigger a migration.

@spalger suggested hashing each plugin's mappings, and storing / diffing those hashes. Early on, we abandoned the hashing approach because we thought it'd trigger migrations too often (e.g. in situations where all we really need to do is patch the index mappings (such as adding a new field to a mapping). The hash approach will trigger a full migration on any change, unless we employ some complicated tricks.

I think we should move forward with a hash-based mappings diff, though, unless someone can think of a better approach that is robust to mapping properties being inserted by ES internals or broadly scoped index templates.