updateMappingOnMaster never times out leaving replicas stuck in INITIALIZING

[folke]

Due to issue #9065, tasks of priority NORMAL never got executed.

A problem related to this is that we had a number of replicas that stayed stuck in INITIALIZING in the phase TRANSLOG.

This was due to the fact that the task "recovery_mapping_check" never actually completed (priority NORMAL). In the method updateMappingOnMaster there is a timeout, but that timeout is never reached due to the latch.await(); code above the timeout check.

Version: 1.4.2

[folke]

Relevant code: https://github.com/elasticsearch/elasticsearch/blob/635ae29bf1eb3aa83597bc296ff58a5b95ca648a/src/main/java/org/elasticsearch/indices/recovery/ShardRecoveryHandler.java

[bleskes]

@folke sorry for the very late response. The latch you refer to waits on the execution of recovery_mapping_check on the local node. The bug you mentioned concerns (if I understand correctly) refreshing the mapping on the master. Can you confirm it kept causing cluster state publishing?
I think a timeout in the recovery_mapping_check makes sense - waiting for the confirmation to see it will actually solve the issue.

[bleskes]

this was resolved with #9575, forgot to close at the time.