Write genesis file in state migration

[alecps]

Rebase of #167 on celo7 branch with some edits to support baklava and mainnet in addition to alfajores.

• Output the genesis json to --outfile.genesis path. Nodes will need this in order to sync.
• During the migration some hardforks are enabled. To make syncing work, they need to be added to the config stored in the genesis file so that the forkids match
• Write genesis allocs to new db for forward compatibility

Other changes:

• Fixes an unrelated error in the pre-migration where getStrayAncients and migrateAncients were competing for a lock on the oldDB freezer.

Tested:

• Manually / locally on alfajores datadir

Fixes #159

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.81%. Comparing base (6e1861d) to head (f9c39bb).

Additional details and impacted files

@@           Coverage Diff           @@
##            celo8     #219   +/-   ##
=======================================
  Coverage   80.81%   80.81%           
=======================================
  Files          32       32           
  Lines        2466     2466           
=======================================
  Hits         1993     1993           
  Misses        396      396           
  Partials       77       77           

Flag	Coverage Δ
cannon-go-tests	80.81% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[alecps]

@piersy @palango Okay I've fixed two issues since the last round of reviews.

(1) Some extra fields were getting added to the genesis.json that we are now deleting. These fields are part of the Genesis type in the current code base but were not present in the legacy Genesis type. They were getting set to null / zero values but this was changing the hash.
(2) We were reading the exraData field from block 0 stored in the non-ancient db. This is a duplicate record of the genesis block to the one stored in ancients and is only accessed when we open a db instance without access to ancients. This is what we were doing in the state migration function before writing the genesis.json, so we were getting an extraData field there that was not transformed and did not match the one in Dango.

After these changes, the diff between the Dango genesis.json posted here and the genesis.json output by the migration script is shown below.
You'll see that the only remaining diff is in the hardfork block numbers and timestamps, which is to be expected as the Dango migration was run much earlier than the test migration that generated the genesis.json using the changes in this PR.

[alecps]

Waiting to merge into celo8 @karlb

[piersy]

Why not use backticks here, as we did for the other two allocs?

[jcortejoso]

Can we update this PR to celo8 as base? I've run a test on an archive node but get this error, I think because the rollup.json and genesis.json was generated with a more updated branch:

INFO [09-16|19:02:21.072] Processing Block Range                   process=non-ancients          from=26,211,658 to(inclusve)=26,216,657 count=4999
INFO [09-16|19:02:21.826] TIMER                                    process=migrateNonAncientsDb  duration=31.248367644s
INFO [09-16|19:02:21.827] TIMER                                    process=removeBlocks          duration="409.547µs"
INFO [09-16|19:02:21.827] Removed stray ancient blocks still in leveldb process=non-ancients          removedBlocks=36
INFO [09-16|19:02:21.827] Non-Ancient Block Migration Completed    process=non-ancients          migratedNonAncients=89999
INFO [09-16|19:02:27.991] TIMER                                    process="non-ancient migration" duration=41.846648191s
INFO [09-16|19:02:27.991] State Migration Started                  newDBPath=/output/celo/chaindata_migrated deployConfig=/output/config/config.json l1Deployments=/output/config/deployment-l1.json l1RPC="https://holesky.cel2.celo-networks-dev.org?apikey=69a921736b1a9abcda36d138f39d919c3478d969e770e09e1d614ba7e720a4f4" l2AllocsPath=/output/config/l2-allocs.json outfileRollupConfig=/output/config/rollup.json
INFO [09-16|19:02:28.002] TIMER                                    process="state migration"       duration=10.56922ms
INFO [09-16|19:02:28.002] TIMER                                    process="full migration"        duration=1h58m6.60066968s
CRIT [09-16|19:02:28.003] error in migration                       err="failed to run full migration: failed to run state migration: cannot unmarshal deploy config: json: unknown field \"useAltDA\""
Command exited with non-zero status 1

[karlb]

Can we backport this to celo8?

Yes, the plan is to merge this into celo8 (although I would't use the term "backport" in this case).

[alecps]

@palango @piersy Rebased this onto celo8 but now I'm getting an error. Will have to investigate when I'm back on Thursday

CRIT [09-17|19:28:45.858] error in migration err="failed to run full migration: failed to run state migration: cannot unmarshal deploy config: json: unknown field "usePlasma""

[karlb]

CRIT [09-17|19:28:45.858] error in migration err="failed to run full migration: failed to run state migration: cannot unmarshal deploy config: json: unknown field "usePlasma""

Plasma as been renamed to AltDA, so this might just be in need of a rename.

[jcortejoso]

I think you need to generate the migration config files from this branch (or ~celo8) to work.

I'm testing with these files if you want to use them:

      wget -O /output/config/config.json https://storage.googleapis.com/cel2-rollup-files/alfajores-mvp/config.json
      wget -O /output/config/deployment-l1.json https://storage.googleapis.com/cel2-rollup-files/alfajores-mvp/deployment-l1.json
      wget -O /output/config/l2-allocs.json https://storage.googleapis.com/cel2-rollup-files/alfajores-mvp/l2-allocs.json

[jcortejoso]

Finished successfully from a recent alfajores full data dir and fresh config files (the ones from my previous comment). Full logs.

INFO [09-18|11:28:25.221] Copied account                           address=0x42000000000000000000000000000000000002F6
INFO [09-18|11:28:25.222] Migrated OP contracts into state DB      totalAllocs=2079 copiedAccounts=2064 overwrittenAccounts=2
INFO [09-18|11:28:25.224] Setting up CeloDistributionSchedule balance
INFO [09-18|11:28:25.231] Read contract addresses                  tokenAddress=0xF194afDf50B03e69Bd7D057c1Aa9e10c9954E4C9 distributionScheduleAddress=0x78Af211Ad79bCE6BF636640CE8c2C2b29e02365A
INFO [09-18|11:28:25.232] Set up CeloDistributionSchedule balance  distributionScheduleAddress=0x78Af211Ad79bCE6BF636640CE8c2C2b29e02365A balance=889,746,012,574,949,080,212,044,180 total_supply=110,253,987,425,050,919,787,955,820 ceiling=1,000,000,000,000,000,000,000,000,000
INFO [09-18|11:28:25.233] Committing state DB
INFO [09-18|11:28:26.370] Build Cel2 migration header              header="&{ParentHash:0xc3a663bf1e5f5f2a29a522a7746787d81222aaf323dcf4183ad30cbccd8e8652 UncleHash:0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 Coinbase:0x4200000000000000000000000000000000000011 Root:0x4cfa0514d6b362556090b665b8e096107ed59bc3c75c365e8b16ad2387c0278f TxHash:0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 ReceiptHash:0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 Bloom:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] Difficulty:+0 Number:+26247232 GasLimit:35000000 GasUsed:0 Time:1726658906 Extra:[67 101 108 111 32 76 50 32 109 105 103 114 97 116 105 111 110] MixDigest:0x0000000000000000000000000000000000000000000000000000000000000000 Nonce:[0 0 0 0 0 0 0 0] BaseFee:+25000000000 WithdrawalsHash:0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 BlobGasUsed:0xc035c5f9f8 ExcessBlobGas:0xc035c5fa00 ParentBeaconRoot:0x2813e4233b3cc22cb6a3c031d5b04e4e5e4845eddadb897258fbe1d1a0d2c251}"
INFO [09-18|11:28:26.371] Built Cel2 migration block               hash=f7d0f9..13af9b root=4cfa05..c0278f number=26,247,232
INFO [09-18|11:28:26.371] Committing trie DB
INFO [09-18|11:28:26.465] Persisted trie from memory database      nodes=7993 size=2.21MiB time=94.30249ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [09-18|11:28:26.470] Wrote updated chain config               config="{\"chainId\":44787,\"homesteadBlock\":0,\"daoForkSupport\":true,\"eip150Block\":0,\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"berlinBlock\":26247232,\"londonBlock\":26247232,\"arrowGlacierBlock\":26247232,\"grayGlacierBlock\":26247232,\"mergeNetsplitBlock\":26247232,\"shanghaiTime\":1726658906,\"cancunTime\":1726658906,\"bedrockBlock\":26247232,\"regolithTime\":1726658906,\"canyonTime\":1726658906,\"ecotoneTime\":1726658906,\"fjordTime\":1726658906,\"graniteTime\":1726658906,\"cel2Time\":1726658906,\"gingerbreadBlock\":19814000,\"terminalTotalDifficulty\":0,\"terminalTotalDifficultyPassed\":true,\"optimism\":{\"eip1559Elasticity\":6,\"eip1559Denominator\":50,\"eip1559DenominatorCanyon\":250}}"
INFO [09-18|11:28:26.476] Wrote genesis file for syncing new nodes path=/output/config/genesis.json
INFO [09-18|11:28:26.476] Wrote genesis state spec (alloc) to database
INFO [09-18|11:28:26.477] Wrote CeL2 migration block               height=26,247,232 root=0x4cfa0514d6b362556090b665b8e096107ed59bc3c75c365e8b16ad2387c0278f hash=0xd68a25ced7a2c91e07ff282dcc98eed15a05c7314f8dd94afd5b184255fff54e timestamp=1,726,658,906
INFO [09-18|11:28:26.548] Updated Cel2 state
INFO [09-18|11:28:26.549] Writing rollup config                    file=/output/config/rollup.json
INFO [09-18|11:28:26.549] State Migration Completed
INFO [09-18|11:28:26.549] TIMER                                    process="state migration"       duration=17.039925787s
INFO [09-18|11:28:26.549] Full Migration Finished                  oldDBPath=/output/celo/chaindata newDBPath=/output/celo/chaindata_migrated
INFO [09-18|11:28:26.549] TIMER                                    process="full migration"        duration=31m34.645560461s
INFO [09-18|11:28:26.549] Finished full migration successfully!
real	31m 37.38s
user	17m 7.63s
sys	7m 34.76s

[alecps]

Okay I've also been able to successfully run the migration with the testdata @jcortejoso provided (which I've added to the PR in the last commit)

Gonna go ahead and merge this so it doesn't block anyone