Index throttling should not apply to recovery

[mikemccand]

@boaz thought of this:

When merges are falling too far behind, we now throttle (bottleneck) the incoming indexing to a single thread to let them catch up. It seems to work well in general, and it's very important to keep the index shard healthy (contain segment count) otherwise all sorts of other problems will cascade.

However, during recovery, we replay the indexing operations from the transaction log to catch up, it's dangerous that we also throttle that thread along with ongoing indexing operations, and it could mean recovery takes a very long time.

It seems like we should somehow fix index throttling to give priority to the indexing thread doing recovery?

[bleskes]

Maybe the simplest solution is to allow the operations that come from the translog (via org.elasticsearch.index.shard.IndexShard#performRecoveryOperation ) to be applied without acquiring the throttling lock.

[mikemccand]

Thanks, I'll try that approach @bleskes