Rename processor does not like fieldnames with dots in them

[ycombinator]

I'm not entirely sure if this is a bug or an enhancement or neither (working as designed). So marking this as discuss for now.

Consider this pipeline simulation:

POST _ingest/pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "foo.bar": 17
      }
    }],
    "pipeline": {
      "processors": [
        {
          "rename": {
            "field": "foo.bar",
            "target_field": "baz"
          }
        }
      ]
    }
}

This will return this error response:

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "exception",
            "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [foo.bar] doesn't exist",
            "header" : {
              "processor_type" : "rename"
            }
          }
        ],
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [foo.bar] doesn't exist",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "java.lang.IllegalArgumentException: field [foo.bar] doesn't exist",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "field [foo.bar] doesn't exist"
          }
        },
        "header" : {
          "processor_type" : "rename"
        }
      }
    }
  ]
}

However, if we first expand the foo.bar field using the dot_expander processor, it works as expected:

POST _ingest/pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "foo.bar": 17
      }
    }],
    "pipeline": {
      "processors": [
        {
          "dot_expander": {
            "field": "foo.bar"
          }
        },
        {
          "rename": {
            "field": "foo.bar",
            "target_field": "baz"
          }
        }
      ]
    }
}

Successful response:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "baz" : 17,
          "foo" : { }
        },
        "_ingest" : {
          "timestamp" : "2019-01-15T22:42:20.90721Z"
        }
      }
    }
  ]
}

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

@ycombinator This is expected behavior in the example you provided (for any processor, not just rename). The dot processor exists to help disambiguate (and handle) dot notation vs. nested object notation (the latter is the default)

For example:

      "_source": {
        "foo.bar": 17,
        "foo" : {
          "bar" : 18
        }
      }
    }]

In a processor, when you reference "foo.bar" do you mean the value of 17 or 18 ? With the dot processor (by default) you get an array of both [17,18], with out the dot processor, it will default to only reading the nested object.

Happy to discuss this further, but in the context of the issue logged (rename processor) I will close this issue. Feel free to re-open or open a new issue with ideas on how to improve the experience here.

[ycombinator]

Thanks for the explanation, @jakelandis. The current behavior makes sense, given the ambiguity problem.

One side-effect in terms of user experience is that we end up needing a lot of dot_expander processors, something that could be alleviated by #36950.