change(state): Write finalized blocks to the state in a separate thread, to avoid network and RPC hangs

[teor2345]

Motivation

This PR starts moving state block write requests into their own task.
When it is finished, it will make state block writes fully concurrent.

Part of #4937.
Closes #5125.

Designs

See #4937 and https://docs.google.com/drawings/d/1FXpAUlenDAjl8nkftrypdAPsj0jr-Ut9gZlSP57nuyc/edit

Solution

Add a block write task:

• Commit finalized blocks to the state in a separate task
• Add last_block_hash_sent to the state service, to avoid database accesses when draining the queue
• Add task handles to check for task exits and panics

Add block write channels:

• Finalized state
• Non-finalized state (not used in this PR)
• Close the finalized block channel when we're finished with it
• Remove a redundant block_in_place()

Deal with errors & shutdowns:

• Update last_block_hash_sent regardless of commit errors
• Check for panics in the block write task
• Close state channels and tasks on drop
• Drop channels and check for closed channels in the block commit task
• Work around a RocksDB shutdown bug in the tests, by removing previous code that cancelled RocksDB background work on shutdown
• Drop any leftover queued blocks when finished with the finalized state, and return an error on their channels
• Return an error on the channels of replaced duplicate blocks

Related fixes:

• Wait for the block commit task in tests, and check for errors
• Wait for chain tip updates in the RPC tests
• Improve RPC error logging
• When running a proptest that sleeps a lot, only run it once
• Explain finalized block duplicate handling and consensus rules

Related cleanups:

• Remove some duplicate fields across StateService and ReadStateService
• Rename a field to StateService.max_queued_finalized_height

Testing

We should run a full sync on this before we merge, and make sure it is not significantly slower than the main branch.

Review

This PR is part of regular scheduled work.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

Handle non-finalized blocks.
Handle task exits and panics correctly.
Testing.

[teor2345]

I'd also like to remove Clone from NonFinalizedState, because it doesn't work the same way as FinalizedState.clone().

That's not possible, so I just updated the docs.

[arya2]

Excellent.

[upbqdn]

Looks great. I left one minor q.

[teor2345]

One last fix to the block height check in b150e3f, and I think we're done.