Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple field names/patterns for (path_)(un)match #66364

Closed
EmilBode opened this issue Dec 15, 2020 · 5 comments · Fixed by #95558
Closed

Allow multiple field names/patterns for (path_)(un)match #66364

EmilBode opened this issue Dec 15, 2020 · 5 comments · Fixed by #95558
Assignees
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@EmilBode
Copy link

Use Case
I'd like to have a dynamic mapping, for different fields, that can't all be described in one pattern.
The most straightforward way of doing so would be to provide an array of fieldnames to the match-argument in a dynamic template, see the example.

And if we're going to change this, I think it would be logical to do the same for unmatch and path_match/path_unmatch

Desired behaviour
I'd like to be be able to specify:

PUT testindex
{
  "mappings": {
    "dynamic_templates": [
      {
        "test": {
          "match": ["one*", "*two"],
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

PUT testindex/_doc/1
{
  "one_test": "eentje",
  "test_two": "twee",
  "three": "drie"
}

And have the resulting mappings for testindex :

"properties" : {
  "test_two" : {
    "type" : "keyword"
  },
  "three" : {
    "type" : "text",
    "fields" : {
      "keyword" : {
        "type" : "keyword",
        "ignore_above" : 256
      }
    }
  },
  "one_test" : {
    "type" : "keyword"
  }
}

Current behaviour
Right now, the dynamic template is ignored. Under water, the array is converted to a string-representation, so the dynamic template actually becomes

[
  {
    "test": {
      "match": "[one*, *two]",
      "mapping": {
        "type": "keyword"
      }
    }
  }
]

Which will, of course, not match on any of my fields.

Workaround
It is of course possible to specify each pattern seperately, like below, but this gets large when there are more patterns.
Alternatively, a regex could match all field names (^one.*|.*two$), but this get also get cumbersome.

PUT testindex
{
  "mappings": {
    "dynamic_templates": [
      {
        "test": {
          "match": "one*",
          "mapping": {
            "type": "keyword"
          }
        }
      },
      {
        "test": {
          "match": "*two",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}
@EmilBode EmilBode added >enhancement needs:triage Requires assignment of a team area label labels Dec 15, 2020
@danhermann danhermann added :Search Foundations/Mapping Index mappings, including merging and defining field types and removed needs:triage Requires assignment of a team area label labels Dec 16, 2020
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Dec 16, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@shaigbdb
Copy link

I got here looking for a solution for the "unmatch" scenario, that is the pattern should match everything besides two or more patterns.
The workaround I found for this is based on the one by @EmilBode . This matches everything besides fields starting with one_ or two_ .

    "dynamic_templates": [


            {
              "other_dynamic": {
                "mapping": {
                  "enabled": false,
                  "type": "object"
                },
                "match": "^(?!(one_|two_)).+$",
                  "match_pattern": "regex"
              }
            }
    ]

@ruflin
Copy link
Member

ruflin commented Feb 20, 2023

Would be great to get this moving in the context of elastic/integrations#5055. Here the specific can be found: elastic/elastic-package#1093 Having support for arrays on match and patch_match would make the template much simpler and more readable.

quux00 added a commit to quux00/elasticsearch that referenced this issue Apr 27, 2023
Arrays of patterns are now allowed for dynamic_templates in the match,
unmatch, path_match and path_unmatch fields. DynamicTemplate has been modified to
support List<String> for these fields. The patterns can be either simple wildcards
or regex. As with previous functionality, mixing of wildcards and regex will not
throw an error, but will not work as expected at mapping time.

One new error pathway was added: if a user specifies a list of non-strings for
one of these pattern fields (e.g., "match": [10, false]) a MapperParserException
will be thrown.

Closes elastic#66364.
quux00 added a commit that referenced this issue Apr 27, 2023
…5558)

* Allow multiple field names/patterns for (path_)(un)match (#66364)

Arrays of patterns are now allowed for dynamic_templates in the match,
unmatch, path_match and path_unmatch fields. DynamicTemplate has been modified to
support List<String> for these fields. The patterns can be either simple wildcards
or regex. As with previous functionality, when match_pattern="regex", simple wildcards
will be flagged with an error, but when match_pattern="simple", using regular expressions
in the match will not throw an error.

One new error pathway was added: if a user specifies a list of non-strings for
one of these pattern fields (e.g., "match": [10, false]) a MapperParserException
will be thrown.

A dynamic_template yamlRestTest was added. This is a BWC change, so the REST test
that uses arrays of patterns is limited to v8.9 and above.

Closes #66364.
@ruflin
Copy link
Member

ruflin commented Apr 28, 2023

Great to see this landed in Elasticsearch. @P1llus This should make it possible to simplify the ECS templates but not sure how you will deal with it as it is only available in newer versions of Elasticsearch.

@P1llus
Copy link
Member

P1llus commented Apr 28, 2023

@ruflin the template that is going to be bundled with ES for logs-* should use this then, and we can handle some sort of reference in fleet, that way it can still use the template bundled with an integration for old releases.

@javanna javanna added Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch and removed Team:Search Meta label for search team labels Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants