Consistent update of stage and failure message in IndexShardSnapshotStatus

[tlrx]

As @ywelsch commented in #26480 (comment), IndexShardSnapshotStatus is not consistently updated.

This pull request adds a synchronized updateStage(stage, failure) to ensure that both fields are updated at the same time and also makes the fields volatile so that threads always read the latest values.

Closes #26480

[ywelsch]

This does not fix the issue, unfortunately. Although updateStage is synchronized, access to the volatile fields is still non-synchronized, which means that when we first call stage() and then failure() to read the values, we can still observe a stage which is FAILURE but a failure that is not set. We need not only atomic writes, but also atomic access to the <stage, failure> field pair.

[tlrx]

@ywelsch I've been to quick on this, sorry. I change the stage / failure getter to be synchronized too.

[ywelsch]

This still won't allow the stage & failure pair to be consistently read. Assume the following sequence:

• read stage
• update(stage, failure)
• read failure

[tlrx]

@ywelsch Right again, I'll revisit this correctly and update the code.

[tlrx]

Closed in favor of #28130