Release Zebra to 1.0.0-beta.13 (#4765)

* update zebra to 1.0.0-beta.13 * add changelog * update the release date * Update changelog for cryptographic batch PRs * Add `bitvec` to the changelog * Update CHANGELOG.md * Add zebra-network fix and tokio version bump * Recommend that users update their rust compiler * Update disk and network usage for recent large blocks * Fix spacing * Add disk and network usage update to README * apply size suggestions Co-authored-by: teor <[email protected]> * add known performance issues * update changelog * change release date Co-authored-by: teor <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ZcashFoundation · Jul 30, 2022 · 61f3639 · 61f3639
1 parent 65b0a8b
commit 61f3639
Show file tree

Hide file tree

Showing 17 changed files with 109 additions and 36 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,20 +5,77 @@ All notable changes to Zebra are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org).
 
 
-## Next Release (Draft)
+## [Zebra 1.0.0-beta.13](https://github.com/ZcashFoundation/zebra/releases/tag/v1.0.0-beta.13) - 2022-07-29
 
-This release improves Zebra's sync and verification performance under heavy load.
-(TODO - complete the summary.)
+This release fixes multiple bugs in proof and signature verification, which were causing big performance issues near the blockchain tip.
+It also improves Zebra's sync performance and reliability under heavy load.
+
+### Disk and Network Usage Changes
+
+Zebra now uses around 50 - 100 GB of disk space, because many large transactions were recently added to the block chain. (In the longer term, several hundred GB are likely to be needed.)
+
+When there are a lot of large user-generated transactions on the network, Zebra can upload or download 1 GB or more per day.
 
 ### Configuration Changes
 
-- Split the checkpoint and full verification [`sync` concurrency options](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4726):
+- Split the checkpoint and full verification [`sync` concurrency options](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4726, #4758):
   - Add a new `full_verify_concurrency_limit`
   - Rename `max_concurrent_block_requests` to `download_concurrency_limit`
   - Rename `lookahead_limit` to `checkpoint_verify_concurrency_limit`
   For backwards compatibility, the old names are still accepted as aliases.
+- Add a new `parallel_cpu_threads` [`sync` concurrency option](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4776).
+  This option sets the number of threads to use for CPU-bound tasks, such as proof and signature verification.
+  By default, Zebra uses all available CPU cores.
+
+### Rust Compiler Bug Fixes
+
+- The Rust team has recently [fixed compilation bugs](https://blog.rust-lang.org/2022/07/19/Rust-1.62.1.html) in function coercions of `impl Trait` return types, and `async fn` lifetimes.
+  We recommend that you update your Rust compiler to release 1.62.1, and re-compile Zebra.
+
+### Added
+
+- Add a `rayon` thread pool for CPU-bound tasks (#4776)
+- Run multiple cryptographic batches concurrently within each verifier (#4776)
+- Run deserialization of transaction in a rayon thread (#4801)
+- Run CPU-intensive state updates in parallel rayon threads (#4802)
+- Run CPU-intensive state reads in parallel rayon threads (#4805)
+- Support Tiers and supported platforms per Tier doc (#4773)
+
+### Changed
+
+- Update column family names to match Zebra's database design (#4639)
+- Update Zebra's mainnet and testnet checkpoints (#4777, #4833)
+- Process more blocks and batch items concurrently, so there's a batch ready for each available CPU (#4776)
+- Wrap note commitment trees in an `Arc`, to reduce CPU and memory usage (#4757)
+- Increment `tokio` dependency from 1.19.2 to 1.20.0, to improve diagnostics (#4780)
+
+### Fixed
+
+- Only verify halo2 proofs once per transaction (#4752)
+- Use a separate channel for each cryptographic batch, to avoid dropped batch results (#4750)
+- Improve batch fairness and latency under heavy load (#4750, #4776)
+- Run all verifier cryptography on blocking CPU-bound threads, using `tokio` and `rayon`  (#4750, #4776)
+- Use `tokio`'s `PollSemaphore`, instead of an outdated `Semaphore` impl (#4750)
+- Check batch worker tasks for panics and task termination (#4750, #4777)
+- Limit the length of the `reject` network message's `message` and `reason` fields (#4687)
+- Change the `bitvec` dependency from 1.0.0 to 1.0.1, to fix a performance regression (#4769)
+- Fix an occasional panic when a `zebra-network` connection closes (#4782)
+- Stop panicking when the connection error slot is not set (#4770)
+- When writing blocks to disk, don't block other async tasks (#4199)
+- When sending headers to peers, only deserialize the header data from disk (#4792)
+- Return errors from `send_periodic_heartbeats_with_shutdown_handle` (#4756)
+- Make FindHeaders and FindHashes run concurrently with state updates (#4826)
+- Stop reading redundant blocks for every FindHashes and FindHeaders request (#4825)
+- Generate sapling point outside the method (#4799)
+
+#### CI
+
+- Workaround lightwalletd hangs by waiting until we're near the tip (#4763)
+- Split out Canopy logs into a separate job (#4730)
+- Make full sync go all the way to the tip (#4709)
+- Split Docker logs into sprout, other checkpoints, and full validation (#4704)
+- Add a Zebra cached state update test, fix lightwalletd tests (#4813)
 
-(TODO - insert changelog here)
 
 ## [Zebra 1.0.0-beta.12](https://github.com/ZcashFoundation/zebra/releases/tag/v1.0.0-beta.12) - 2022-06-29
 

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/README.md b/README.md
@@ -76,7 +76,7 @@ for your platform:
 2. Install Zebra's build dependencies:
      - **libclang:** the `libclang`, `libclang-dev`, `llvm`, or `llvm-dev` packages, depending on your package manager
      - **clang** or another C++ compiler: `g++`, `Xcode`, or `MSVC`
-3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.12 zebrad`
+3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.13 zebrad`
 4. Run `zebrad start` (see [Running Zebra](https://zebra.zfnd.org/user/run.html) for more information)
 
 For more detailed instructions, refer to the [documentation](https://zebra.zfnd.org/user/install.html).
@@ -95,7 +95,7 @@ cargo install --features=<name> ...
 The recommended requirements for compiling and running `zebrad` are:
 - 4+ CPU cores
 - 16+ GB RAM
-- 50GB+ available disk space for building binaries and storing finalized state
+- 100 GB+ available disk space for building binaries and storing cached chain state
 - 100+ Mbps network connections
 
 We continuously test that our builds and tests pass on:
@@ -147,8 +147,8 @@ If this is a problem for you, please
 [open a ticket.](https://github.com/ZcashFoundation/zebra/issues/new/choose)
 
 `zebrad`'s typical mainnet network usage is:
-- Initial sync: 31 GB download
-- Ongoing updates: 10-100 MB upload and download per day, depending on peer requests
+- Initial sync: 40 GB download (in the longer term, several hundred GB are likely to be downloaded).
+- Ongoing updates: 10 MB - 1 GB upload and download per day, depending on user-created transaction size, and peer requests
 
 Zebra also performs an initial sync every time its internal database version changes.
 
@@ -182,9 +182,25 @@ So Zebra's state should always be valid, unless your OS or disk hardware is corr
 ## Known Issues
 
 There are a few bugs in Zebra that we're still working on fixing:
-- [No Windows support #3801](https://github.com/ZcashFoundation/zebra/issues/3801)
+- No Windows support [#3801](https://github.com/ZcashFoundation/zebra/issues/3801)
   - We used to test with Windows Server 2019, but not anymore; see issue for details
 
+### Performance
+
+We are working on improving Zebra performance, the following are known issues:
+- Send note commitment and history trees from the non-finalized state to the finalized state [#4824](https://github.com/ZcashFoundation/zebra/issues/4824)
+- Speed up opening the database [#4822](https://github.com/ZcashFoundation/zebra/issues/4822)
+- Revert note commitment and history trees when forking non-finalized chains [#4794](https://github.com/ZcashFoundation/zebra/issues/4794)
+- Store only the first tree state in each identical series of tree states [#4784](https://github.com/ZcashFoundation/zebra/issues/4784)
+
+RPCs might also be slower than they used to be, we need to check:
+- Revert deserializing state transactions in rayon threads [#4831](https://github.com/ZcashFoundation/zebra/issues/4831)
+
+Ongoing investigations:
+- Find out which parts of CommitBlock/CommitFinalizedBlock are slow [#4823](https://github.com/ZcashFoundation/zebra/issues/4823)
+- Mini-Epic: Stop tokio tasks running for a long time and blocking other tasks [#4747](https://github.com/ZcashFoundation/zebra/issues/4747)
+- Investigate busiest tasks per tokio-console [#4583](https://github.com/ZcashFoundation/zebra/issues/4583)
+
 ## Future Work
 
 Features:

diff --git a/book/src/user/install.md b/book/src/user/install.md
@@ -9,7 +9,7 @@ for your platform:
 2. Install Zebra's build dependencies:
      - **libclang:** the `libclang`, `libclang-dev`, `llvm`, or `llvm-dev` packages, depending on your package manager
      - **clang** or another C++ compiler: `g++`, `Xcode`, or `MSVC`
-3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.12 zebrad`
+3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.13 zebrad`
 4. Run `zebrad start` (see [Running Zebra](run.md) for more information)
 
 If you're interested in testing out `zebrad` please feel free, but keep in mind

diff --git a/tower-batch/Cargo.toml b/tower-batch/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "tower-batch"
-version = "0.2.27"
+version = "0.2.28"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT"
 edition = "2021"

diff --git a/tower-fallback/Cargo.toml b/tower-fallback/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "tower-fallback"
-version = "0.2.27"
+version = "0.2.28"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT"
 edition = "2021"

diff --git a/zebra-chain/Cargo.toml b/zebra-chain/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-chain"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-consensus/Cargo.toml b/zebra-consensus/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-consensus"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-network/Cargo.toml b/zebra-network/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-network"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-network/src/constants.rs b/zebra-network/src/constants.rs
@@ -242,7 +242,7 @@ pub const TIMESTAMP_TRUNCATION_SECONDS: u32 = 30 * 60;
 /// [BIP 14]: https://github.com/bitcoin/bips/blob/master/bip-0014.mediawiki
 //
 // TODO: generate this from crate metadata (#2375)
-pub const USER_AGENT: &str = "/Zebra:1.0.0-beta.12/";
+pub const USER_AGENT: &str = "/Zebra:1.0.0-beta.13/";
 
 /// The Zcash network protocol version implemented by this crate, and advertised
 /// during connection setup.

diff --git a/zebra-node-services/Cargo.toml b/zebra-node-services/Cargo.toml
@@ -2,7 +2,7 @@
 name = "zebra-node-services"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 edition = "2021"
 repository = "https://github.com/ZcashFoundation/zebra"
 

diff --git a/zebra-rpc/Cargo.toml b/zebra-rpc/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-rpc"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-script/Cargo.toml b/zebra-script/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-script"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-state/Cargo.toml b/zebra-state/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-state"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-test/Cargo.toml b/zebra-test/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "zebra-test"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
 edition = "2021"

diff --git a/zebra-utils/Cargo.toml b/zebra-utils/Cargo.toml
@@ -2,7 +2,7 @@
 name = "zebra-utils"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 edition = "2021"
 # Prevent accidental publication of this utility crate.
 publish = false

diff --git a/zebrad/Cargo.toml b/zebrad/Cargo.toml
@@ -2,7 +2,7 @@
 name = "zebrad"
 authors = ["Zcash Foundation <[email protected]>"]
 license = "MIT OR Apache-2.0"
-version = "1.0.0-beta.12"
+version = "1.0.0-beta.13"
 edition = "2021"
 # Zebra is only supported on the latest stable Rust version. Some earlier versions might work.
 # Zebra uses features introduced in Rust 1.58.