feat: track side effects in public

[alexghr]

This PR adds start/end side effect counters to private & public context inputs.

This solves #4958 by assigning the correct side effect counters to notes created in setup/teardown so that when they're recombined into a single array they can be sorted in order of creation. A few e2e tests were also added to validate the change.

Next steps:

• track public storage reads and writes as side effects in Noir (currently done in parallel, in TS)
• validate a call's side effects are in between its start and end counters in the private and kernels
• when calling a public function, the public context should update its context.side_effect_counter to the called function's end_side_effect_counter + 1 (just like the private context)
• rename call_context.side_effect_counter to just call_context.counter

Fix #4958
Fix #5185

[AztecBot]

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 94922fc2 and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	5,668	18,820	36,356
l1_rollup_calldata_gas	66,016	238,672	469,676
l1_rollup_execution_gas	666,648	953,533	1,336,984
l2_block_processing_time_in_ms	1,461 (+14%)	5,013 (+6%)	8,977 (+1%)
note_successful_decrypting_time_in_ms	170 (+20%)	524 (-7%)	968 (+3%)
note_trial_decrypting_time_in_ms	73.5 (+59%)	40.1 (-50%)	100 (+32%)
l2_block_building_time_in_ms	17,222 (+2%)	68,099 (+1%)	135,744 (+1%)
l2_block_rollup_simulation_time_in_ms	7,031 (+1%)	27,870	55,725
l2_block_public_tx_process_time_in_ms	10,167 (+2%)	40,171 (+2%)	79,918 (+2%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	14,721 (+4%)	26,358 (-1%)
note_history_successful_decrypting_time_in_ms	1,224 (-5%)	2,415 (-2%)
note_history_trial_decrypting_time_in_ms	60.5 (-42%)	122 (-22%)
node_database_size_in_bytes	19,075,152	35,709,008
pxe_database_size_in_bytes	29,859	59,414

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	284 (+2%)	44,370	27,700
private-kernel-ordering	217	52,324	14,326
base-rollup	633	177,083	925
root-rollup	68.1	4,176	789
private-kernel-inner	637 (+2%)	73,229	27,700
public-kernel-app-logic	448 (+2%)	35,262	28,215
public-kernel-tail	176	40,926	28,215
merge-rollup	8.26	2,696	925

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves	32 leaves
batch_insert_into_append_only_tree_16_depth_ms	9.92	15.9	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.6	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.579	0.494	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	45.6 (-1%)	71.8	230	442	868 (+1%)	1,704 (-1%)	N/A
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	96.0	159	543	1,055	2,079	4,127	N/A
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.469 (-1%)	0.444	0.420	0.415	0.413 (+1%)	0.409 (-1%)	N/A
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	53.3 (-1%)	105 (-1%)	336 (+1%)	656	1,302	2,587	N/A
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	104	207	691	1,363	2,707	5,395	N/A
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.476	0.476	0.457	0.452	0.453	0.450	N/A
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	60.7
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	109
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.532

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes
tx_size_in_bytes	22,014

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	3,991 (+1%)	1,737 (+1%)

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	10.8 (-12%)	1,251 (+2%)

[github-actions]

Changes to circuit sizes

Generated at commit: e900cd5a0abd0f08ae20b1825f007c5e10483bfa, compared to commit: 94922fc24e728100b456ed5f0203974964fd9f83

🧾 Summary (100% most significant diffs)

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
public_kernel_app_logic	+14 ❌	+0.03%	+1,201 ❌	+0.38%
public_kernel_setup	+4 ❌	+0.01%	+625 ❌	+0.35%
public_kernel_teardown	+4 ❌	+0.01%	+625 ❌	+0.34%
private_kernel_inner	+1 ❌	+0.00%	+312 ❌	+0.06%
private_kernel_init	+1 ❌	+0.00%	+2 ❌	+0.00%

---

Full diff report 👇

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
public_kernel_app_logic	55,535 (+14)	+0.03%	314,516 (+1,201)	+0.38%
public_kernel_setup	27,611 (+4)	+0.01%	180,301 (+625)	+0.35%
public_kernel_teardown	29,357 (+4)	+0.01%	182,048 (+625)	+0.34%
private_kernel_inner	95,969 (+1)	+0.00%	522,008 (+312)	+0.06%
private_kernel_init	48,686 (+1)	+0.00%	350,744 (+2)	+0.00%

[just-mitch]

Changes seem okay, but I'm definitely a bit hazy on the interplay among the different counters. Docs here (or in the code) would be really helpful.

[just-mitch]

Can you link an issue for fixing end === 0? I believe its what you're referring to in future work in your comment in the PR.

[alexghr]

Oh the comment isn't that great. What I'm saying is that public functions enqueued from private become CallRequests where start_side_effect_counter is the time that they were enqueued and end_side_effect_counter is 0 (they haven't run) while public calls in public execution are fully formed.

[just-mitch]

Why did we need to add this here instead of continuing to use the one in CallContext? I'm struggling a bit with how they are different, and why the one in CallContext got renamed to just side_effect_counter.

[alexghr]

@LeilaWang came with the suggestion to remove it from CallContext entirely, which I've attempted to do in #5177

Why did we need to add this here instead of continuing to use the one in CallContext?

The value in call_context is either the start_side_effect_counter for private calls and public calls in public execution, while it is the enqueued_side_effect_counter for enqueued public calls.

The real start_side_effect_counter is the one returned by context.finish() and for enqueued calls it will be different than what's in call_context (I enqueue it at 5 but run the function at 20). Because of the hashing that goes on in the kernel I can't modify call_context.side_effect_counter for enqueued calls.

In #5177 we remove call_context.side_effect_counter and rely on public inputs.

[LeilaWang]

🚀