Move network transaction deserialization to a dedicated blocking and CPU-heavy thread

[teor2345]

Motivation

Zebra's sync can be slow or fail, because transaction deserialization involves some CPU-heavy cryptography.

So we want to move transaction deserialization into a rayon CPU thread pool scope.

API Reference

Blocking threads:
https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html

CPU-heavy threads:
https://docs.rs/rayon/latest/rayon/fn.scope_fifo.html

Detailed Analysis

See #4583 (comment)

Designs

When zebra-network deserializes a block or transaction, move that work to the rayon thread pool.

Example code for rayon in tokio blocking threads:

zebra/zebra-consensus/src/primitives/ed25519.rs

Lines 121 to 134 in 81727d7

	/// Flush the batch using a thread pool, and return the result via the channel.
	/// This function returns a future that becomes ready when the batch is completed.
	fn flush_spawning(batch: BatchVerifier, tx: Sender) -> impl Future<Output = ()> {
	// Correctness: Do CPU-intensive work on a dedicated thread, to avoid blocking other futures.
	tokio::task::spawn_blocking(|| {
	// TODO:
	// - spawn batches so rayon executes them in FIFO order
	// possible implementation: return a closure in a Future,
	// then run it using scope_fifo() in the worker task,
	// limiting the number of concurrent batches to the number of rayon threads
	rayon::scope_fifo(|s| s.spawn_fifo(|_s| Self::verify(batch, tx)))
	})
	.map(|join_result| join_result.expect("panic in ed25519 batch verifier"))
	}

Related Work

State deserialization is handled in #4788