Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check for duplicates in template generation #17203

Closed
urso opened this issue Mar 24, 2020 · 2 comments · Fixed by #18849
Closed

Check for duplicates in template generation #17203

urso opened this issue Mar 24, 2020 · 2 comments · Fixed by #18849
Assignees
Labels
enhancement libbeat Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@urso
Copy link

urso commented Mar 24, 2020

Having multiple dynamic_templates with the same field name in templates can lead to bugs. For example: elastic/apm-server#2693, elastic/apm-server#2693 (comment)

The Elasticsearch team is discussing to disallow multiple dynamic templates: elastic/elasticsearch#53326

Beats template generator should check for duplicates and fail (or at least warn) if we detect duplicates.

@urso urso added libbeat Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team labels Mar 24, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@simitt
Copy link
Contributor

simitt commented Mar 24, 2020

In APM Server we currently have the same key in the dynamic template for e.g. labels:

    "mappings" : {
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "type" : "keyword"
            },
            "match_mapping_type" : "string"
          }
        },
        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "type" : "boolean"
            },
            "match_mapping_type" : "boolean"
          }
        },
        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "scaling_factor" : 1000000,
              "type" : "scaled_float"
            },
            "match_mapping_type" : "*"
          }
        },
        ...
     ]
  }

The logic for creating the dynamic template with multiple entries for the same key in libbeat could be adapted to e.g. adding a sequence to the key if the same key exists multiple times.
The corresponding definition in _meta/fields.yml looks like the following

    - name: labels
      type: object
      object_type_params:
        - object_type: keyword
        - object_type: boolean
        - object_type: scaled_float
          scaling_factor: 1000000
      dynamic: true

@simitt simitt self-assigned this May 29, 2020
simitt added a commit to simitt/beats that referenced this issue May 29, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes elastic#17203
simitt added a commit that referenced this issue Jun 2, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes #17203
simitt added a commit to simitt/beats that referenced this issue Jun 2, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes elastic#17203
simitt added a commit that referenced this issue Jun 2, 2020
When multiple dynamic templates are detected for the same field, add the
matching type to the template name for uniqueness.

closes #17203
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement libbeat Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants