VReplication: retry in WaitForPos when read of pos is killed off by deadlock detector

[mattlord]

Description

For whatever reason(s), we've seen the deadlock detector fire more frequently when using MySQL 8.0 in our CI tests. This has, in particular, hit our VReplication related tests where there is very high contention on a small number of _vt.vreplication records. The record(s) in the table are getting read by different connections (by id, which is the Primary Key) while also getting constantly updated in other connections to update the progress: pos, rows_copied, heartbeat, etc.

This has led to the VReplication related tests that use MySQL 8.0 to be flaky, in particular:

• onlineddl_vrepl_stress_mysql80
• vreplication_across_db_versions

In this PR we retry the read of the _vt.vreplication.pos field if we get a deadlock detected error when waiting for a tablet to reach a replication position, until we hit the context timeout.

Related Issue(s)

Fixes: #10590
Related-to: #10620

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests are not required
• Documentation is not required

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
• If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.
• If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.

[shlomi-noach]

Looks good! Thank you for investigating this!

One thing, since calls to dbClient.ExecuteFetch can go into loop, is that we must now check ctx.Done() or ctx.Err() before calling qr, err := dbClient.ExecuteFetch(binlogplayer.ReadVReplicationStatus(uint32(id)), 10). Which requires some delicate refactoring because we then cannot populate the error message with data from qr.

[mattlord]

Looks good! Thank you for investigating this!

One thing, since calls to dbClient.ExecuteFetch can go into loop, is that we must now check ctx.Done() or ctx.Err() before calling qr, err := dbClient.ExecuteFetch(binlogplayer.ReadVReplicationStatus(uint32(id)), 10). Which requires some delicate refactoring because we then cannot populate the error message with data from qr.

Good point! If we continually hit the deadlock error then it's possible that we won't get to the select where we read from the context's done channel... I'll think about this.

[mattlord]

Looks good! Thank you for investigating this!
One thing, since calls to dbClient.ExecuteFetch can go into loop, is that we must now check ctx.Done() or ctx.Err() before calling qr, err := dbClient.ExecuteFetch(binlogplayer.ReadVReplicationStatus(uint32(id)), 10). Which requires some delicate refactoring because we then cannot populate the error message with data from qr.

Good point! If we continually hit the deadlock error then it's possible that we won't get to the select where we read from the context's done channel... I'll think about this.

@shlomi-noach done here: cb01ba8

Thanks again!

[mattlord]

Well, the DCO check flakiness has forced me to run the CI tests quite a few times. Good news is that it has demonstrated that the tests using MySQL 8.0 are now very stable! 🙂They would often need to be run 2 or 3 times to pass, now they're passing on the first try. 🥳