Reindex wait_for_completion=false task failure

[jbudz]

curl -XPUT "http://localhost:9220/foo"
curl -XPOST "http://localhost:9220/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d'{  "source": {    "index": "foo"  },  "dest": {    "index": "bar"  }}'

// run twice
curl -XPOST "http://localhost:9220/_reindex?wait_for_completion=false" -H 'Content-Type: application/json' -d'{  "source": {    "index": "foo"  },  "dest": {    "index": "bar"  }}'

Error log

   │ info [o.e.t.LoggingTaskListener] [windesk] 6563 failed with exception
   │      org.elasticsearch.index.mapper.MapperParsingException: Root mapping definition has unsupported parameters:  [task : {_meta={version=2}, dynamic=strict, properties={completed={type=boolean}, task={properties={action={type=keyword}, cancellable={type=boolean}, id={type=long}, parent_task_id={type=keyword}, node={type=keyword}, running_time_in_nanos={type=long}, start_time_in_millis={type=long}, type={type=keyword}, status={type=object, enabled=false}, description={type=text}, headers={type=object, enabled=false}}}, response={type=object, enabled=false}, error={type=object, enabled=false}}}]
   │            at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:135) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:83) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
   │            at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:553) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

The response code is 200 with a task id. When attempting to retrieve the returned task id:

curl -XGET "http://localhost:9220/_tasks/UWqGoA9jRM25A2GQIUZp3g:6563"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "resource_not_found_exception",
        "reason" : "task [UWqGoA9jRM25A2GQIUZp3g:6563] isn't running and hasn't stored its results"
      }
    ],
    "type" : "resource_not_found_exception",
    "reason" : "task [UWqGoA9jRM25A2GQIUZp3g:6563] isn't running and hasn't stored its results"
  },
  "status" : 404
}

Re-indexing fails until the .tasks index is deleted, and then I can run it once before it starts failing again.

[jbudz]

Edited the title to clarify, minimal reproduction with empty index but happens with docs too.

[jimczi]

It looks like the creation of the .tasks index fails due to an issue with the mapping. It might be a leftover of the type removal but I am surprised that it's not caught by tests. @romseygeek can you take a look ?

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Reindex)

[romseygeek]

The inbuilt tasks mapping still has a top-level type of tasks, instead of _doc, which is what's causing the failure. The fix is simple, but I'm trying to work out why tests haven't picked this up so far.

[tylersmalley]

Thanks @romseygeek - we're happy to help test the fix to help unblock some downstream teams since we're currently pinning Kibana to a previous snapshot.

[romseygeek]

This is a combination of two problems, the tasks default mapping and the fact that we haven't incremented the version metadata in the mapping either, so it gets re-applied with every new task. I'll open a PR to fix master, and also a 7x backport to fix the version.