2024‐01‐02 Testnet Rollback and Restart

Edit

As of 2024-01-03 16:46 UTC testnet is back online. The instructions below are no longer relevant, but you will need to update your shred version and bank hash and start normally:

--expected-shred-version 14676 \
--expected-bank-hash 2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi \

---

This testnet restart is NOT urgent. Follow these instructions when you have time, but don’t skip sleep or disrupt other plans for this.

Summary

In order to test upgrading from v1.16 to v1.17 we would like to roll testnet back to the same version and feature gates as mainnet-beta. In order to roll back feature gates we need to shut testnet down and restart it from snapshots.

Attribute	Value
Validator version	v1.16.24
Snapshot slot	244_604_255
Restart slot	244_604_256
Shred version	14676
Expected bank hash	2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi

Step 1. Stop validator process if you haven’t already

Step 2. Create snapshot

This command creates a snapshot but removes the activated v1.16 & v1.17 feature gate accounts that won’t be activated on mainnet-beta before v1.17 is recommended. Deleting these accounts will revert testnet back to the same feature gates that mainnet will have when it is updated. If you would like to verify this list of accounts the keys are in feature_set.rs and you can check which features are enabled on each cluster with solana feature status -u[m|t]

solana-ledger-tool --ledger <ledger-path> create-snapshot \
--snapshot-archive-path  <snapshot-path> \
--hard-fork 244604255 \
--deactivate-feature-gate \
16FMCmgLzCNNz6eTwGanbyN2ZxvTBSLuQ6DZhgeMshg \
25vqsfjk7Nv1prsQJmA4Xu1bN61s8LXCBGUPp8Rfy1UF \
5ZCcFAzJ1zsFKe1KSZa9K92jhx7gkcKj97ci2DBo1vwj \
5wAGiy15X1Jb2hkHnPDCM8oB9V42VNA9ftNVFK84dEgv \
5x3825XS7M2A3Ekbn5VGGkvFoAg5qrRWkTrY4bARP1GL \
7axKe5BTYBDD87ftzWbk5DfzWMGyRvqmWTduuo22Yaqy \
8pgXCMNXC8qyEFypuwpXyRxLXZdpM4Qo72gJ6k87A6wL \
A16q37opZdQMCbe5qJ6xpBB9usykfv8jZaMkxvZQi4GJ \
BTWmtJC8U5ZLMbBUUA1k6As62sYjPEjAiNAT55xYGdJU \
Cdkc8PPTeTNUPoZEfCY5AyetUrEdkZtNPMgz58nqyaHD \
D2aip4BBr8NPWtU9vLrwrBvbuaQ8w1zV38zFLxx4pfBV \
D31EFnLgdiysi84Woo3of4JMu7VmasUS3Z7j9HYXCeLY \
FKAcEvNgSY79RpqsPNUV5gDyumopH4cEHqUxyfm8b8Ap \
GwtDQBghCTBgmX2cpEGNPxTEBUTQRaDMGTr5qychdGMj \
--  244604255 <snapshot-path>

The output should include this at (or near) the end:

Successfully created snapshot for slot 244604256, hash 2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi: /home/sol/ledger-snapshots-epoch-579/snapshot-244604256-G8h6stNYLsCia1NVC3fLGnsEMgQPR5wGCgSNEu2PZA6D.tar.zst
Shred version: 14676

Once you have created a snapshot move all the other snapshots to a backup directory, so your snapshot directory contains only

snapshot-244604256-G8h6stNYLsCia1NVC3fLGnsEMgQPR5wGCgSNEu2PZA6D.tar.zst

If you get an error like this:

Error: Slot 244604255 is not available

Or this:

Unable to process blockstore from starting slot <slot> to 244604255; the ending slot is less than the starting slot. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found.

Your snapshots directory contains a snapshot that is for a slot >244604255. If you also have a snapshot for slot <=244604255 then move snapshots for slots >244604255 to a backup directory and run the solana-ledger-tool command again. If you do not have a snapshot for slot <=244604255 then proceed to the appendix for instructions on downloading a snapshot.

If you are unable to create a snapshot proceed to the appendix for instructions on downloading a snapshot.

Step 3: Install version v1.16.24

Important, make sure to downgrade your software to v1.16.24

solana-install init v1.16.24

Step 4: Start your validator

Add these arguments to your validator startup script:

--wait-for-supermajority 244604256 \
--expected-shred-version 14676 \
--expected-bank-hash 2ZHZpzSpBhkbfqsENGybfLbXSZ2hZiTq79qHCM4TWBpi \

If you get an error like this delete your ledger/rocksdb directory and restart again.

solana_ledger::blockstore] Opening database at <ledger rocksdb path>
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RocksDb(Error { message: "Invalid argument: Column families not 
opened: merkle_root_meta" })', core/src/validator.rs:1978:92

As it starts, the validator will load the snapshot for slot 244604256 and wait for 80% of the stake to come online before producing/validating new blocks.

To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting:

INFO  solana_core::validator] Waiting for 80% of activated stake at slot 244604256 to be in gossip...

And if you have RPC enabled, ask it repeated for the current slot:

solana --url http://127.0.0.1:8899 slot

Any number other than 244604256 means you did not complete the steps correctly.

Once started you should see log entries for “active stake” visible in gossip and “waiting for 80% of stake” to be visible. You can track these to see how the stake progresses.

---

Appendix (use this only if step 3 failed)

If you couldn’t produce your snapshot locally follow these appendix steps

Step 1: Download a snapshot from a known validator

If you are unable to generate a snapshot locally for slot 244604256 you will need to download one from a known validator. Add these lines to your startup script.

--known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
--expected-shred-version 14676 \

Remove the flag --no-snapshot-fetch in your startup script if it is present.

Step 2: After download, restart

Verify that you have a new snapshot in your snapshot directory. If the snapshot is done downloading, stop your validator process.

Add the flag --no-snapshot-fetch to your startup script

Start your validator again using the instructions from step 4 above.