429s due to replica operations

[bcastilho90]

I have a cluster with 3 master nodes and several data nodes. From time to time, we experience a brief period where certain nodes will return 429s due to replica operations:

es_rejected_execution_exception Reason: "rejected execution of primary operation [coordinating_and_primary_bytes=0, replica_bytes=2210080256, all_bytes=2210080256, primary_operation_bytes=30430, max_coordinating_and_primary_bytes=2147483648

I've scaled the cluster before, but we continue to see this intermittently since it happens on random nodes. Is there a way to have the readiness probe fail on cases like this so that requests stop being sent to a node that is overloaded?

I'm not sure how exactly to find out what is causing the back log of replica ops.