[Epic] Scaling the provers 📈

[alexghr]

We need a scalable and resilient model for handling the proof necessary to build large blocks at a high frequency.

Relevant links

• Engineering design: A proving job broker implementation
• HackMD: Performant job pool
• Excalidraw

Design

Beta Give feedback

1. Merge engineering design
2. Design new Broker interface for both consumers and producers
3. Mock out architecture (e.g. when using our ProvingBroker vs established queuing system)

Tasks

Beta Give feedback

1. [Prover] New ProvingBroker #9529
   team-alpha +1
   
2. [Prover] ProvingBroker restores persisted jobs and proofs #9530
   team-alpha +1
   
3. [Prover] ProvingBroker resolves conflicts when same jobs is worked on by two agents #9531
   team-alpha +1
   
4. [Prover] Persist proving jobs #9532
   team-alpha +1
   
5. [Prover] Updates to ProvingAgent to work with new ProvingBroker #9533
   team-alpha +1
   
6. [Prover] add k8s files for ProvingBroker and small/big ProvingAgent #10108
   +0
   
7. [Prover][Docs] Update docs with info about broker/orchestrator storage requirements #10109
   +0

Tests

Beta Give feedback

1. [Prover][Test] Ensure proper integration between orchestrator<->broker<->agent (no serialisation errors) #9537
   team-alpha +1
   
2. [Prover][Test] Performance test one broker + 1000 agents #9538
   team-alpha +1
   
3. [Provier][Test] Performance test: one broker + 10k agents #9539
   team-alpha +1
   
4. [Prover][Test] Broker failure: system must recover if the ProvingBroker goes down #9540
   team-alpha +1
   
5. [Prover][Test] Agent failure: a busy agent goes down, the broker must give their job to another agent #9541
   team-alpha +1
   
6. [Prover][Test] Agent temporary failure: agent goes down temporarily #9542
   team-alpha +1