feat: Multithreaded prover folding

[codygunton]

We only get an 80% reduction on 16 threads, but it's still a nice improvement of overall time.

Before

Benchmarking lock created at ~/BENCHMARK_IN_PROGRESS.
client_ivc_bench                                                              100%   15MB  48.4MB/s   00:00    
2024-03-12T03:38:23+00:00
Running ./client_ivc_bench
Run on (16 X 3000 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.00, 0.00, 0.11
--------------------------------------------------------------------------------
Benchmark                      Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------
ClientIVCBench/Full/6      29446 ms        24872 ms            1 Decider::construct_proof=1 Decider::construct_proof(t)=753.082M ECCVMComposer::compute_commitment_key=1 ECCVMComposer::compute_commitment_key(t)=3.78026M ECCVMComposer::compute_witness=1 ECCVMComposer::compute_witness(t)=1.73739G ECCVMComposer::create_prover=1 ECCVMComposer::create_prover(t)=3.40893G ECCVMComposer::create_proving_key=1 ECCVMComposer::create_proving_key(t)=1.66752G ECCVMProver::construct_proof=1 ECCVMProver::construct_proof(t)=1.76066G Goblin::merge=11 Goblin::merge(t)=128.627M GoblinTranslatorCircuitBuilder::constructor=1 GoblinTranslatorCircuitBuilder::constructor(t)=56.5845M GoblinTranslatorComposer::create_prover=1 GoblinTranslatorComposer::create_prover(t)=123.279M GoblinTranslatorProver::construct_proof=1 GoblinTranslatorProver::construct_proof(t)=927.524M ProtoGalaxyProver_::accumulator_update_round=10 ProtoGalaxyProver_::accumulator_update_round(t)=3.46156G ProtoGalaxyProver_::combiner_quotient_round=10 ProtoGalaxyProver_::combiner_quotient_round(t)=7.17713G ProtoGalaxyProver_::perturbator_round=10 ProtoGalaxyProver_::perturbator_round(t)=1.38221G ProtoGalaxyProver_::preparation_round=10 ProtoGalaxyProver_::preparation_round(t)=4.1G ProtogalaxyProver::fold_instances=10 ProtogalaxyProver::fold_instances(t)=16.1209G ProverInstance(Circuit&)=11 ProverInstance(Circuit&)(t)=1.945G batch_mul_with_endomorphism=30 batch_mul_with_endomorphism(t)=562.528M commit=425 commit(t)=3.96966G compute_combiner=10 compute_combiner(t)=7.175G compute_perturbator=9 compute_perturbator(t)=1.38188G compute_univariate=48 compute_univariate(t)=1.41821G construct_circuits=6 construct_circuits(t)=4.20217G
Benchmarking lock deleted.
client_ivc_bench.json                                                         100% 4015   130.4KB/s   00:00    
function                                        ms     % sum
construct_circuits(t)                         4202    14.31%
ProverInstance(Circuit&)(t)                   1945     6.62%
ProtogalaxyProver::fold_instances(t)         16121    54.89%
Decider::construct_proof(t)                    753     2.56%
ECCVMComposer::create_prover(t)               3409    11.61%
GoblinTranslatorComposer::create_prover(t)     123     0.42%
ECCVMProver::construct_proof(t)               1761     5.99%
GoblinTranslatorProver::construct_proof(t)     928     3.16%
Goblin::merge(t)                               129     0.44%

Total time accounted for: 29370ms/29446ms = 99.74%

Major contributors:
function                                        ms    % sum
commit(t)                                     3970   13.52%
compute_combiner(t)                           7175   24.43%
compute_perturbator(t)                        1382    4.71%
compute_univariate(t)                         1418    4.83%

Breakdown of ECCVMProver::create_prover:
ECCVMComposer::compute_witness(t)             1737    50.97%
ECCVMComposer::create_proving_key(t)          1668    48.92%

Breakdown of ProtogalaxyProver::fold_instances:
ProtoGalaxyProver_::preparation_round(t)           4100    25.43%
ProtoGalaxyProver_::perturbator_round(t)           1382     8.57%
ProtoGalaxyProver_::combiner_quotient_round(t)     7177    44.52%
ProtoGalaxyProver_::accumulator_update_round(t)    3462    21.47%

After

Benchmarking lock created at ~/BENCHMARK_IN_PROGRESS.
client_ivc_bench                                                                                                                                                                                                                                            100%   15MB  53.8MB/s   00:00    
2024-03-12T13:10:44+00:00
Running ./client_ivc_bench
Run on (16 X 3606.49 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 36608 KiB (x1)
Load Average: 0.00, 0.92, 1.95
--------------------------------------------------------------------------------
Benchmark                      Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------
ClientIVCBench/Full/6      26935 ms        22185 ms            1 Decider::construct_proof=1 Decider::construct_proof(t)=760.055M ECCVMComposer::compute_commitment_key=1 ECCVMComposer::compute_commitment_key(t)=3.6897M ECCVMComposer::compute_witness=1 ECCVMComposer::compute_witness(t)=1.74049G ECCVMComposer::create_prover=1 ECCVMComposer::create_prover(t)=3.41363G ECCVMComposer::create_proving_key=1 ECCVMComposer::create_proving_key(t)=1.66922G ECCVMProver::construct_proof=1 ECCVMProver::construct_proof(t)=1.76915G Goblin::merge=11 Goblin::merge(t)=128.378M GoblinTranslatorCircuitBuilder::constructor=1 GoblinTranslatorCircuitBuilder::constructor(t)=57.5506M GoblinTranslatorComposer::create_prover=1 GoblinTranslatorComposer::create_prover(t)=121.602M GoblinTranslatorProver::construct_proof=1 GoblinTranslatorProver::construct_proof(t)=927.714M ProtoGalaxyProver_::accumulator_update_round=10 ProtoGalaxyProver_::accumulator_update_round(t)=722M ProtoGalaxyProver_::combiner_quotient_round=10 ProtoGalaxyProver_::combiner_quotient_round(t)=7.24835G ProtoGalaxyProver_::perturbator_round=10 ProtoGalaxyProver_::perturbator_round(t)=1.39394G ProtoGalaxyProver_::preparation_round=10 ProtoGalaxyProver_::preparation_round(t)=4.11739G ProtogalaxyProver::fold_instances=10 ProtogalaxyProver::fold_instances(t)=13.4817G ProverInstance(Circuit&)=11 ProverInstance(Circuit&)(t)=2.0017G batch_mul_with_endomorphism=30 batch_mul_with_endomorphism(t)=564.354M commit=425 commit(t)=3.98668G compute_combiner=10 compute_combiner(t)=7.24616G compute_perturbator=9 compute_perturbator(t)=1.3936G compute_univariate=48 compute_univariate(t)=1.42762G construct_circuits=6 construct_circuits(t)=4.25115G
Benchmarking lock deleted.
client_ivc_bench.json                                                                                                                                                                                                                                       100% 4021   139.1KB/s   00:00    
function                                        ms     % sum
construct_circuits(t)                         4251    15.83%
ProverInstance(Circuit&)(t)                   2002     7.45%
ProtogalaxyProver::fold_instances(t)         13482    50.20%
Decider::construct_proof(t)                    760     2.83%
ECCVMComposer::create_prover(t)               3414    12.71%
GoblinTranslatorComposer::create_prover(t)     122     0.45%
ECCVMProver::construct_proof(t)               1769     6.59%
GoblinTranslatorProver::construct_proof(t)     928     3.45%
Goblin::merge(t)                               128     0.48%

Total time accounted for: 26855ms/26935ms = 99.70%

Major contributors:
function                                        ms    % sum
commit(t)                                     3987   14.85%
compute_combiner(t)                           7246   26.98%
compute_perturbator(t)                        1394    5.19%
compute_univariate(t)                         1428    5.32%

Breakdown of ECCVMProver::create_prover:
ECCVMComposer::compute_witness(t)             1740    50.99%
ECCVMComposer::create_proving_key(t)          1669    48.90%

Breakdown of ProtogalaxyProver::fold_instances:
ProtoGalaxyProver_::preparation_round(t)           4117    30.54%
ProtoGalaxyProver_::perturbator_round(t)           1394    10.34%
ProtoGalaxyProver_::combiner_quotient_round(t)     7248    53.76%
ProtoGalaxyProver_::accumulator_update_round(t)     722     5.36%

[AztecBot]

Benchmark results

Metrics with a significant change:

• node_history_sync_time_in_ms (10): 30,357 (+16%)
• note_trial_decrypting_time_in_ms (32): 97.3 (+109%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 58e15edf and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	5,668	18,820	36,356
l1_rollup_calldata_gas	65,980	238,492	468,500
l1_rollup_execution_gas	666,612	953,353	1,335,808
l2_block_processing_time_in_ms	1,279 (-2%)	4,814	9,172 (+1%)
note_successful_decrypting_time_in_ms	149 (+6%)	579 (+9%)	946 (-1%)
note_trial_decrypting_time_in_ms	54.1 (+23%)	⚠️ 97.3 (+109%)	79.1 (-10%)
l2_block_building_time_in_ms	16,998 (-1%)	67,638	135,938 (+1%)
l2_block_rollup_simulation_time_in_ms	7,025 (-1%)	27,636 (-2%)	55,608 (-1%)
l2_block_public_tx_process_time_in_ms	9,948 (-1%)	39,945 (+1%)	80,227 (+1%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	14,407 (+3%)	⚠️ 30,357 (+16%)
note_history_successful_decrypting_time_in_ms	1,239 (+5%)	2,517 (+5%)
note_history_trial_decrypting_time_in_ms	91.0 (+36%)	193 (+58%)
node_database_size_in_bytes	19,062,864	35,676,240
pxe_database_size_in_bytes	29,859	59,414

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	275 (-1%)	44,338	27,700
private-kernel-ordering	215 (-1%)	52,324	14,326
base-rollup	636	177,083	925
root-rollup	67.9 (-1%)	4,176	789
private-kernel-inner	627 (-1%)	73,197	27,700
public-kernel-app-logic	441	35,198	28,215
public-kernel-tail	178 (+1%)	40,926	28,215
merge-rollup	8.26	2,696	925

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves	32 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.0	16.0 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.6	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.585	0.494 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	45.8	72.0	230	443	861 (-1%)	1,715	N/A
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	96.0	159	543	1,055	2,079	4,127	N/A
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.471	0.445	0.419	0.415	0.411	0.411	N/A
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	53.5	106	337 (+1%)	656	1,301 (-1%)	2,591 (-1%)	N/A
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	104	207	691	1,363	2,707	5,395	N/A
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.478	0.480	0.459 (+1%)	0.453	0.452 (-1%)	0.452 (-1%)	N/A
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	60.8 (-1%)
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	109
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.533

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes
tx_size_in_bytes	22,014

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	3,976	1,718 (-1%)

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	13.5 (+4%)	1,235

[maramihali]

Nice!