change(state): Check database format is valid every 5 minutes, to catch format errors in new block code

[teor2345]

Motivation

We want to catch format errors in new block code. But currently that code is only tested when Zebra adds new blocks, then shuts down, then opens the same state. So PRs can merge with bugs in that code.

Instead, we check the format is valid every 5 minutes while Zebra is running.

This is the periodic part of #7570.

Complex Code or Requirements

We can't run this code too often, or it will impact performance. But we want to make sure it gets run during most tests, so we catch bugs early.

Solution

Continue running the format change thread after the initial change finishes, then periodically run a format check:

• Turn the format change task into a continuous loop
• Move the max height check into the format checks
• Make slower format checks cancellable on shutdown

Related changes:

• Refactor the format change/check code so that "no change" has an enum variant
• Add a variant for checking the format of new blocks
• Refactor the format checks so they can be called using a single method
• Make all the format checks run completely, and log their errors, then panic
• Time format upgrades and format checks
• Store the format change flag in the database struct for debugging

Example Logs

On startup, it takes 2 seconds to open the database, and 7 seconds to check the format is valid:

2023-09-21T02:45:10.692975Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open newer database format: data should be compatible running_version=25.2.1 disk_version=25.2.2
2023-09-21T02:45:12.388329Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as downgraded running_version=25.2.1 disk_version=25.2.2 
2023-09-21T02:45:19.485921Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=25.2.1 inital_disk_version=25.2.2

So the format check is running approximately 2% of the time. Since it runs in parallel with the rest of the code, and it will be faster when the data is in the database memory cache, this is acceptable performace.

Review

This is on the critical path for finishing off the subtree work. It's not strictly required for the 1.3.0 release, but it would help to find any other data bugs.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

[mpguerra]

Does this run on CI only or on any running zebra instance?

[teor2345]

Does this run on CI only or on any running zebra instance?

This runs on all Zebra instances, because if the database format is wrong, then we can break a consensus rule, or send incorrect data to wallets.

(We already run these checks at startup, but that's not enough to detect all format bugs.)