Unable to initialize Fleet - incorrect mappings applied to internal transform indices - WITH FIX

[deepybee]

Kibana version: 7.12.0

Elasticsearch version: 7.12.0

Server OS version: Ubuntu 18.04.5 LTS

Browser version: Google Chrome Version 89.0.4389.114 (Official Build) (x86_64)

Browser OS version: macOS Catalina 10.15.7

Original install method (e.g. download page, yum, from source, etc.): apt

Describe the bug: After upgrade from an earlier version (presumed 7.11.x but I am uncertain the last time I accessed Fleet on this cluster) Fleet fails with the following error. I also hit #90984 and it looks like the same bug regarding ordering of upgrades may well have caused this too.

This symptom is caused by a background transform index being created with default mappings. When Fleet tries to start it attempts to execute the transform but Transforms cannot load the transform document because it expects id to be a keyword but per default mapping it is indexed as text with a subfield of .keyword which Transforms neither expects to be present nor attempts to parse.

NB: I have already implemented the workaround in that issue which is probably a dependency to ensuring a working Fleet once the workaround here is actioned.

Steps to reproduce:

1. Upgrade Elasticsearch and Kibana to 7.12.0 from an earlier version
2. Log into Kibana
3. Open Fleet

Expected behavior:

Fleet opens and no errors are thrown

Screenshots (if relevant):

Errors in browser console (if relevant): 400 Bad Request error with JSON payload as per above screenshot

Provide logs and/or server output (if relevant):

Any additional context:

Buried in that response is the offending index which is - at least on this cluster - .transform-internal-005. This index has picked up a default mapping somewhere and actually breaks Transforms completely as well as Fleet:

GET _cat/transforms?v

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".transform-internal-005",
        "node" : "aM8jyf3eQqKJ9q1qHxkGuA",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
      }
    }
  },
  "status" : 400
}

Checking the mapping on the reported index confirms it is using a default mapping:

GET .transform-internal-005/_mapping

#! this request accesses system indices: [.transform-internal-005], but in a future major version, direct access to system indices will be prevented by default
{
  ".transform-internal-005" : {
    "mappings" : {
      "properties" : {
        "create_time" : {
          "type" : "long"
        },
        "description" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "dest" : {
          "properties" : {
            "index" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },

...

Fix

NB This workaround requires at least 1 node in the cluster to have the transform role

1. Use a reindex to backup the erroneously mapped index's data

POST _reindex
{
  "source": {
    "index": ".transform-internal-005"
  },
  "dest": {
    "index": "broken-transform-data-backup"
  }
}

2. Delete the broken index

DELETE .transform-internal-005

3. The Transforms page should now load correctly and show the updated transform version (0.18.0 on Stack version 7.12)
   

4. Reload the Fleet page. After a few moments the background transform will have finished and Fleet will load as expected:

7. If all went well, delete the backup.

DELETE broken-transform-data-backup

[EricDavisX]

@kevinlog and @nchaulet it looks like some upgrade problems can still be hit wrt to the Transforms.

[kevinlog]

@EricDavisX this is another set up which required that at least node is set to handle transforms. We've got a DOCS ticket (7.13) to more clearly document this requirement: elastic/security-docs#608

In addition, this PR: #95649 will add more finer grained error in Fleet setup

[elasticmachine]

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[EricDavisX]

I explicitly thought that @deepybee was saying the fix of having a node setup didn't work. I may have mis-understood - glad to be, if that is the case.

[deepybee]

@EricDavisX The fix / workaround definitely does work, but it has a dependency on there being a node in the cluster that has the transform role.

In practice, this should present as a fairly low risk footprint as any small uniform cluster will have the role implicitly enabled on all nodes, meaning the note is only applicable to environments where the cluster has nodes with roles explicitly set and none of them have transform in node.roles.

[deepybee]

In fact, now I think about it it would be a very small footprint as I guess it would require a cluster where the transform role was actually removed after having been present for an earlier version of Fleet to work in the first place (which is an edge case I hit on this test cluster).

Aside from migrating from uniform to dedicated node types, I can't think of any scenario likely to affect a live cluster atm.

[EricDavisX]

That's great news - thanks for the extra notes and thought @deepybee . Cheers

[jlind23]

Closing as not relevant anymore. Feel free to reopen if needed