Using same key in dynamic template leads to overwrites

[simioa]

Hi,

I don't know if this is rather a general elasticsearch issue but I found this "bug" in the apm-server Index template.

The dynamic_templates section in the exported Index Template contains objects with the same key which leads to overwrites rather than appends in the final Index Mapping.

The apm-server Index template contains the following definitions when exported with the apm-server export template command. Used version here is 6.8.2

        {
          "context.tags": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string",
            "path_match": "context.tags.*"
          }
        },
        {
          "context.tags": {
            "mapping": {
              "type": "boolean"
            },
            "match_mapping_type": "boolean",
            "path_match": "context.tags.*"
          }
        },
        {
          "context.tags": {
            "mapping": {
              "scaling_factor": 1000000,
              "type": "scaled_float"
            },
            "match_mapping_type": "*",
            "path_match": "context.tags.*"
          }
        },

Those three definitions share the same "context.tags" key
When deployed and a Index is created with this template, only the last "context.tags" definition is contained in the Index Mapping, which leads to indexing failures when the field value is not a float

Renaming the keys to unique values as a workaround seems to fix this.

In the 7.x Index Template context.tags is replaced by labels which seems to have the same issue (not tested)

[simitt]

Hi @simioa, the entries in the dynamic mapping ensure that whenever a new key nested under context.tags (<7.0) or labels (>= 7.0) is indexed, the proper type is derived:
keyword for strings, boolean for booleans, scaled_float otherwise.

I just briefly tested everything through and cannot find an issue. E.g. indexing something like:

{
  "context.tags": {
    "a": true,
    "b": "some string",
    "c": 1234.56,
    "d": 190
  }
}

results in following mapping (derived by GET apm-*/_mapping):

{
  "apm-6.8.4-span" : {
    "mappings" : {
      "doc" : {
        "dynamic_templates" : [
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "match_mapping_type" : "boolean",
              "mapping" : {
                "type" : "boolean"
              }
            }
          },
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "scaling_factor" : 1000000,
                "type" : "scaled_float"
              }
            }
          },
...
        ],
...
        "properties" : {
...
              "tags" : {
                "dynamic" : "true",
                "properties" : {
                  "a" : {
                    "type" : "boolean"
                  },
                  "b" : {
                    "type" : "keyword"
                  },
                  "c" : {
                    "type" : "scaled_float",
                    "scaling_factor" : 1000000.0
                  },
                  "d" : {
                    "type" : "scaled_float",
                    "scaling_factor" : 1000000.0
                  }
                }
              },
...
          }
        }
      }
    }
  }
}

[simioa]

Hi @simitt I applied the original template and made some testing:

GET _template/apm-server-6.8.2-dyntest:

{
  "apm-server-6.8.2-dyntest" : {
    "order" : 10,
    "index_patterns" : [
      "apmdyntest-6.8.2-*"
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "30",
        "number_of_shards" : "2",
        "query" : {
          "default_field" : [
            ...
          ]
        }
      }
    },
    "mappings" : {
      "doc" : {
        "_meta" : {
          "version" : "6.8.2"
        },
        "dynamic_templates" : [
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "type" : "boolean"
              },
              "match_mapping_type" : "boolean"
            }
          },
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "scaling_factor" : 1000000,
                "type" : "scaled_float"
              },
              "match_mapping_type" : "*"
            }
          },
          {
            "fields" : {
              "path_match" : "fields.*",
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string"
            }
          }
...

I then tried to Index the following Event:
POST apmdyntest-6.8.2-test2/doc:

{
    "beat": {
        "hostname": "redacted",
        "version": "6.8.2",
        "name": "redacted"
    },
    "context": {
        "system": {
            "hostname": "redacted",
            "architecture": "amd64",
            "ip": "10.10.10.10",
            "platform": "Linux"
        },
        "process": {
            "pid": 12345,
            "title": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/jre/bin/java"
        },
        "service": {
            "name": "redacted",
            "language": {
                "version": "1.8.0_222",
                "name": "Java"
            },
            "runtime": {
                "version": "1.8.0_222",
                "name": "Java"
            },
            "agent": {
                "version": "1.6.1",
                "name": "java"
            }
        },
        "tags": {
            "name": "PS Scavenge"
        }
    },
    "@version": "1",
    "jvm": {
        "gc": {
            "count": 1626,
            "time": 22172
        }
    },
    "processor": {
        "event": "metric",
        "name": "metric"
    },
    "host": {
        "name": "redacted"
    },
    "@timestamp": "2019-09-11T05:30:46.127Z",
    "tags": [
        "read-from-dlq"
    ]
}

result:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [context.tags.name] of type [scaled_float] in document with id 'GAXgHm0BRxlIJzx3H9VZ'"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [context.tags.name] of type [scaled_float] in document with id 'GAXgHm0BRxlIJzx3H9VZ'",
    "caused_by": {
      "type": "number_format_exception",
      "reason": "For input string: \"PS Scavenge\""
    }
  },
  "status": 400
}

Index Mapping after tried Indexing:
GET apmdyntest-6.8.2-test2/_mapping

{
  "apmdyntest-6.8.2-test2" : {
    "mappings" : {
      "doc" : {
        "_meta" : {
          "version" : "6.8.2"
        },
        "dynamic_templates" : [
          {
            "context.tags" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "scaling_factor" : 1000000,
                "type" : "scaled_float"
              }
            }
          },
          {
            "fields" : {
              "path_match" : "fields.*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
...

Only the scaled_float context.tags dynamic template is present in the mapping.

Replacing

        {
          "context.tags": {
            "path_match": "context.tags.*",
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        },
        {
          "context.tags": {
            "path_match": "context.tags.*",
            "mapping": {
              "type": "boolean"
            },
            "match_mapping_type": "boolean"
          }
        },
        {
          "context.tags": {
            "path_match": "context.tags.*",
            "mapping": {
              "scaling_factor": 1000000,
              "type": "scaled_float"
            },
            "match_mapping_type": "*"
          }
        },

with

        {
          "context.tags1": {
            "path_match": "context.tags.*",
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        },
        {
          "context.tags2": {
            "path_match": "context.tags.*",
            "mapping": {
              "type": "boolean"
            },
            "match_mapping_type": "boolean"
          }
        },
        {
          "context.tags3": {
            "path_match": "context.tags.*",
            "mapping": {
              "scaling_factor": 1000000,
              "type": "scaled_float"
            },
            "match_mapping_type": "*"
          }
        },

leads to successful Indexing (Index apmdyntest-6.8.2-test3) with mapping:
GET apmdyntest-6.8.2-test3/_mapping

        "dynamic_templates" : [
          {
            "context.tags1" : {
              "path_match" : "context.tags.*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "context.tags2" : {
              "path_match" : "context.tags.*",
              "match_mapping_type" : "boolean",
              "mapping" : {
                "type" : "boolean"
              }
            }
          },
          {
            "context.tags3" : {
              "path_match" : "context.tags.*",
              "mapping" : {
                "scaling_factor" : 1000000,
                "type" : "scaled_float"
              }
            }
          },
          {
            "fields" : {
              "path_match" : "fields.*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
...

[simitt]

I just tested with your example document on a fresh 6.8.2 installation and also for the exact version everything worked just fine.

One thing I noticed is that your template shows "order" : 10, which is not the default value when loading with APM Server. Is it possible you have multiple templates matching your index? If so, it could be the root cause for your issues. If multiple templates are matching, they are merged based on the order parameter.

[simioa]

Yes we have a default template(order: 0) for all indices but it does not contain a dynamic_template entry for "context.tags".
I made some further testing, and it seems like merging multiple templates removes duplicate keys from the dynamic_template mapping.
This also happens when you merge the apm-server template with a template containing an empty dynamic_template mapping like this:

{
  "empty_mapping" : {
    "mappings" : {
      "doc" : {
        "_meta" : {
          "version" : "6.8.2"
        },
        "dynamic_templates" : [
        ],
        "properties" : {
...

Is this intended behavior?

[simitt]

Sorry for the late response, with this additional information I was able to reproduce the issue. I am investigating internally how this should be fixed.

[simitt]

@simioa this seems to be an issue with how template keys with the same name are merged. I added a comment in the according ES issue elastic/elasticsearch#29200.

[cbuescher]

Fyi, the ES team discussed the problems arising from template keys with the same name in dynamic_templates and would like to deprecate and eventually remove its use. I opened an issue (elastic/elasticsearch#52062) that summarized the current discussion.

[simitt]

depends on elastic/beats#18849