change(state): Prepare for in-place database format upgrades, but don't make any format changes yet

[teor2345]

Motivation

We need a way to upgrade the on-disk database format, without re-syncing the whole chain.

Close #6955

Complex Code or Requirements

This is concurrent code, we need to be careful to shut down correctly, and regularly check for panics in the inner thread.

A lot of this code is modified from the block write task. We could refactor that task to use a generic ThreadHandle type in a future PR.

Solution

Implementation:

• Move format upgrades to their own module and enum
• Launch a format change thread if needed
• Shut down the format change thread when Zebra is shutting down
• Regularly check for panics in the state upgrade task
• Increment database format to 25.0.2

Logging:

• Log the running and initial disk database format versions on startup

Testing

zebrad tests:

• Test that a newly created state is the current version, and re-opening it doesn't upgrade or downgrade the version
• Test that a fake older state gets upgraded to the current version
• Test that a fake newer state gets downgraded to the current version
• Add extra test support functions and rename some existing functions

CI:

• Add initial disk and running state versions to cached state images in CI
  • This also tests zebrad logging the state versions
• Add a force_save_to_disk argument to the CI workflow

Related changes:

• Add some TODOs and remove a redundant timer

Review

This is a large PR, so I'm happy to do a video review, or split the test harness refactor and the CI changes into separate PRs.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

See #6955:

• documentation for format upgrades
• optional format upgrade debugging log file
• optional refactors

[teor2345]

I'm running a manual test of the workflow changes here:
ZcashFoundation/zebra/actions/runs/5502224802

This worked and created a disk with the correct version description and labels:

[arya2]

Looks good to me!

Thank you for the well-organized tests and related cleanups.

[teor2345]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated