[Meta Issue] Replace Badger in Deal Making with CARv2 and the Sharded DAG Store and test with MRA

[aarshkshah1992]

This is a meta-issue to track the work of replacing Badger from Filecoin deal making with CARv2 and the Sharded DAG Store and do some solid testing along with the MRA changes to ensure a smooth landing.

---

Pending CARv2 features

• CARv2 read-write blockstore should support resumption.

---

Sharded DAG Store

• TODO.

---

Markets

• Integrate CARv2 Blockstore with Storage Markets go-fil-markets#558.
• Integrate CARv2 blockstore in the retrieval market go-fil-markets#560.
• Integrate Sharded DAG Store into the Storage Market.
• Integrate Sharded DAG Store into the Retrieval Market.

---

Lotus

• integrate DAG store and CARv2 in deal-making #6671 (PR is against the MRA changes).
• Integrate the Sharded DAG Store in the Lotus PR above.

---

Migration

• Migration of existing storage client data imports in Badger to CARv2 files:

• Iterate over all the imported payloads in the storage client's Badger store and migrate them to CARv2 files so clients don't have to re-import them. Maybe don't clean up the Badger store right away so client can downgrade without losing imports.

• Register Shards for all active storage deals on the Miner with the Sharded DAG Store. add migration to register deals as shards go-fil-markets#570

• Iterate over all active storage deals on a Miner and register shards for all of them with the Sharded DAG Store so we can serve retrievals for deals made before the upgrade.

• Raise PR against integrate DAG store and CARv2 in deal-making #6671.

---

### Testground testing (MRA + CARv2 changes)

• Upgraded client -> upgraded miner (Online storage deal and retrieval).
• Upgraded client -> upgraded miner (Offline storage deal and retrieval).
• Upgraded client-> old miner (Online storage deal and retrieval).
• Upgraded client-> old miner (Offline storage deal and retrieval).
• Old client -> upgraded miner (Online storage deal and retrieval).
• Old client -> upgraded miner (Offline storage deal and retrieval).

---

Manual testing on Mainnet (MRA + CARv2 changes)

• Upgraded client -> upgraded miner (Online storage deal and retrieval).
• Upgraded client -> upgraded miner (Offline storage deal and retrieval).
• Upgraded client-> old miner (Online storage deal and retrieval).
• Upgraded client-> old miner (Offline storage deal and retrieval).
• Old client -> upgraded miner (Online storage deal and retrieval).
• Old client -> upgraded miner (Offline storage deal and retrieval).
• Old client makes an online storage deal with an old miner -> client upgrades and does a retrieval.
• Old client makes an online storage deal with an old miner -> miner upgrades and then the old client does a retrieval.

---

Benchmarking (MRA + CARv2 changes)

• Benchmark storage deal data transfers of the upgrade vs the use of Badger using our Testground tests.
• Benchmark retrieval deal data transfers of the upgrade vs the use of Badger using our Testground tests.

=> tracking in #7106.

---

Testing in the Wild Wild West (MRA + CARv2 changes)

• Get MinerX fellows to upgrade to the latest changes (will likely need to work with product for this).
• Ask Dealbot and Estuary (nicely) to upgrade and stress-test storage and retrieval deals against Minxer fellows

[aarshkshah1992]

cc @jacobheun @raulk for tracking.

[raulk]

Merged in #6873.