Limit RTG exceptions to retry on

[pxsalehi]

Limit RTG exceptions to retry on and do not throw in getCurrentNodeOfPrimary.
Follow up to #104579 (comment).

Relates ES-5727

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[pxsalehi]

I think i will also open a test PR to show that realtime-(m)get on unassigned shard will/should fail, to have that behaviour documented. But merging this first, would help with the mget PRs too. Sorry the whole get/mget stateless/stateful dimension results in too many PRs.

[pxsalehi]

serverless check failure is not related and has an open issue.

[henningandersen]

Can we add a test that fails without this? I think a test that stops the node containing the shard could do so. Fine if that is in stateless only, but did not see a PR there.

[henningandersen]

I think i will also open a test PR to show that realtime-(m)get on unassigned shard will/should fail, to have that behaviour documented. But merging this first, would help with the mget PRs too. Sorry the whole get/mget stateless/stateful dimension results in too many PRs.

Sorry, missed this, I could be ok with that, but then we need to validate when writing the unassigned test that it failed with the original version of this. Feels a bit backwards and I would prefer to have the failing test first if possible, but can be persuaded 🙂

[ywangd]

LGTM

[pxsalehi]

Feels a bit backwards

It is a bit! :) I'm just trying to reduce all the pending PRs. I can open the PR first if you want. I have a local version of it. It takes me a bit to polish it.

[pxsalehi]

ok. thanks to @ywangd 's quick reviews and help, I've been unblocked differently by merging the mget PRs. This can then wait for the test PR first.

[pxsalehi]

failure was #103588

[pxsalehi]

Oh! I didn't want to merge this actually. My bad! 🤦🏼
I think there are more ways that getCurrentNodeOfPrimary could throw and I didn't cover all of them in the PR. I'll open yet another follow up PR!