[TS | LIP-164000] Reset Sweep Progress

[jeremyk-91]

Goals (and why):

• Have the ability to reset sweep progress. This may be useful to deal with circumstances where entries are written to the targeted sweep queue after the sweep timestamp has progressed past it - while the transaction in question will necessarily fail, there may still be cruft in the targeted sweep queue.

Implementation Description (bullets):

• Add a config flag. If set, on startup does a CAS to 0 and doesn't perform targeted sweep. This is needed because a service has multiple nodes, but once the last node rolls, assuming it starts successfully, no one is running sweep and we CAS the progress to 0 -> we have 0 for all shards and strategies.

Testing (What was existing testing like? What have you done to improve it?): Added some unit tests.

Concerns (what feedback would you like?):

• What happens if the number of shards in the config is not what's actually persisted in the DB? I believe we do this correctly, but worth a check.
• Have I thought through the concurrency model (especially across multiple nodes) correctly? and/or, is this too heavy-handed?

Where should we start reviewing?: ShardProgress

Priority (whenever / two weeks / yesterday): this week

[changelog-app]

Generate changelog in changelog/@unreleased

Type

• Feature
• Improvement
• Fix
• Break
• Deprecation
• Manual task
• Migration

Description

Targeted sweep progress may be reset with the resetTargetedSweepQueueProgressAndStopSweep flag in targeted sweep install configuration. This may be useful in cleaning up cruft in the targeted sweep queue that may have been written by failed transactions.

As the name suggests, this will prevent sweep from cleaning up old cells, so users should not run with this configuration in the steady state. If running your service in HA, once the last node rolls and reports that it has successfully reset the sweep progress table, we can be certain that progress has been reset to zero.

Check the box to generate changelog(s)

• Generate changelog entry

[gmaretic]

It's fine, but I'm not really sure why we want to stop sweep in addition to resetting. Though it's a good forcing function to make people remove the flag so I guess that's the reason.

[gmaretic]

Eh, not a fan of the code duplication, but it's different enough to make it awkward to reuse. We could just delete the entry to make this much simpler but I assume we want HA -- not that sweep will work if we don't have delete consistency anyway...

[gmaretic]

Is this really what we want? We could just allow it to start sweeping. Also the nodes on old versions will still be attempting to sweep anyway

[jeremyk-91]

Discussed offline: this centers around the behaviour of targeted sweep where nodes CAS the bound from (thing I read -> my progress) repeatedly. We need to wait for all nodes to report that they're done with this.

[gmaretic]

Probably want to verify that it only throws once the value actually repeats

[jeremyk-91]

Hmm, I think this is tested by stopsTryingToResetIfSomeoneElseDid() above - in practice you could check the CheckAndSetException's actual values but we don't get that here because of mocks, so I don't see another way of easily doing this?

[gmaretic]

OH I meant just verify that the method was called 4 times

[jeremyk-91]

added!

[gmaretic]

Discussed offline: document why we need to stop running sweep and wait for all nodes to reset

[svc-autorelease]

Released 0.302.5