[Transform] mapper cannot be changed from type [long] to [float]

[przemekwitek]

It happens that the latest transform fails with the following error:

      "reason" : "Failed to index documents into destination index due to permanent error [org.elasticsearch.xpack.transform.transforms.BulkIndexingException: Bulk index experienced [1] failures and at least 1 irrecoverable [mapper [products.min_price] cannot be changed from type [long] to [float]]. Other failures:; java.lang.IllegalArgumentException: mapper [products.min_price] cannot be changed from type [long] to [float]]",

Steps to reproduce:

1. Start Kibana
2. Load kibana_sample_data_ecommerce dataset
3. Go to "Dev Tools"
4. Create a transform:

PUT _transform/debug_mappings
{
  "source": {
    "index": [ "kibana_sample_data_ecommerce" ]
  },
  "dest": {
    "index": "ecommerce_dest"
  },
  "latest": {
    "unique_key": [ "category.keyword" ],
    "sort": "order_date"
  }
}

5. Start the transform:

POST _transform/debug_mappings/_start

6. Get transform stats:

GET _transform/debug_mappings/_stats

The reason for this error is that the products.min_price mapped type in the source index is half_float but the field value is sometimes an integer.
Then, due to dynamic mappings in the destination index, the products.min_price gets mapped as long and cannot be further changed from long to float.

Workaround:
Create destination index mappings explicitly (at least for this one problematic field):

PUT ecommerce_dest
{
  "mappings": {
    "properties": {
      "products": {
        "properties": {
          "min_price": {
            "type": "float"
          }
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[Winterflower]

I can replicate this error on the kibana_sample_data_ecommerce dataset and I'm checking the other sample datasets to see if same thing happens. I'll open a ticket in the Kibana repo about updating these datasets.

[sophiec20]

This behaviour is expected as the latest transform relies on dynamic mappings when creating the destination index. For many use cases, we expect a relevant index template to already exist.

For clarity, this issue is raised as a bug as we would ideally like for a better user experience when evaluating and creating transforms using the Kibana sample data - without the user sometimes needing to create templates or to copy mappings prior to starting the transform.

(In a future version, we may be able to include logic into the UI workflow for managing mappings in instances where a template does not already exist.)

[Winterflower]

The location of the ecommerce sample dataset in the Kibana repo is here

kibana/src/plugins/home/server/services/sample_data/data_sets/ecommerce

There is a file ecommerce.json.gz that contains the data that needs to be modified.
Once you have modified this, submit a PR like this to the Kibana repo elastic/kibana#36791

[przemekwitek]

Thanks for the pointer @Winterflower, I'll give it a try.

[przemekwitek]

The dataset has been fixed so this error should not occur anymore.

[pmunshi1]

we are again getting same error on elastic search and for all float parameters, we are getting below error one after another

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "mapper [programs.RevenueCurrentMonth] cannot be changed from type [float] to [long]"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "mapper [programs.RevenueCurrentMonth] cannot be changed from type [float] to [long]"
    },
    "status": 400
}```

version :"8.2.0"

any help would be appreciated. :) 

[przemekwitek]

A couple of questions, @pmunshi1:

1. What is the mapping type of programs.RevenueCurrentMonth field in the source index?
2. What is the transform config?
3. Does the destination index specified in the transform config already exists? Or you let the transform create it? If it exists, what is the destination index mapping?