RPC unit test timeout fix

[bw-solana]

Problem

rpc_subscriptions::tests::test_check_account_subscribe fails sporadically in CI with a receiver timeout.

This is caused by a race condition between completing processing of the subscribed notification and processing bank transaction. We need the subscription to fully process so they will be notified of the bank transaction, but occasionally (~10% of the time) the bank transaction completes first because these activities are handled in separate threads.

Note this is essentially the same thing observed in #27480 and partially mitigated by #27481

Proposed Solution

Add a short delay in the test_check_account_subscribe test to allow for the subscribed notification to be handled before proceeding with processing the bank transaction.

[sakridge]

Possible to poll here on the subscribe notification getting processed?

[sakridge]

Would be better than naked sleep.

[bw-solana]

I don't believe there's existing instrumentation we can poll to check for completion (but I would love to be wrong!).

We can certainly add the instrumentation, but I'm not crazy about it. It adds a fair amount of complexity for something we would only use for unit tests

[CriesofCarrots]

I think the assert_subscribed method is about the best we have. I wonder if there's some ordering in subscription creation that should be tweaked so that we only pass that assert when everything is completed.
Brennan, could you tell where in the subscription we get to when the tx is processed, in the failure cases?

[bw-solana]

I'm not sure exactly how far we get, but I know we didn't make it to the NotificationEntry::Subscribed type match in fn process_notifications because we didn't hit the print I placed there. I'm sure it varies run-to-run due to the nature of being async.

The assert only ensures completion of the part that is serialized with respect to bank transaction, so it doesn't provide much of anything

[CriesofCarrots]

Thanks. For sure it will vary, I was just curious what you saw.
Yeah, I am thinking if we can rework the subscription registration so that the map entry checked by the assert is the last thing that happens, we could make the assert actually useful. I'll take a look at some point, and we can make this test more robust, if that approach works.
Appreciate you following up on this!

[bw-solana]

Thanks. For sure it will vary, I was just curious what you saw.

Yeah, I am thinking if we can rework the subscription registration so that the map entry checked by the assert is the last thing that happens, we could make the assert actually useful. I'll take a look at some point, and we can make this test more robust, if that approach works.

Appreciate you following up on this!

Sounds good! Thank you!