fix: reprocess notes in pxe when a new contract is added #3867

just-mitch · 2024-01-07T20:12:06Z

Add notion of a Deferred Note: a note that we can decrypt, but not decode because we don't have the contract artifact in our PXE database.

We hang on to these deferred notes in a new store in our PXE database, and reprocess them when their contract is made available.

When we reprocess and successfully decode, we also need to search if we have already nullified the newly decoded note.

AztecBot · 2024-01-07T20:35:14Z

Benchmark results

Metrics with a significant change:

note_history_successful_decrypting_time_in_ms (5): 2,461 (-16%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 5441753e and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	128 txs
l1_rollup_calldata_size_in_bytes	45,444	179,588	716,132
l1_rollup_calldata_gas	222,828	867,896	3,448,220
l1_rollup_execution_gas	936,161	3,970,559	24,791,945
l2_block_processing_time_in_ms	1,328 (+1%)	5,101	21,332 (+2%)
note_successful_decrypting_time_in_ms	324 (-8%)	1,004 (-9%)	3,649 (-12%)
note_trial_decrypting_time_in_ms	33.0 (-49%)	27.3 (-24%)	142 (+5%)
l2_block_building_time_in_ms	14,921	59,404	238,111
l2_block_rollup_simulation_time_in_ms	10,592	42,287	169,874
l2_block_public_tx_process_time_in_ms	4,297	17,030	67,966

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	15,408 (+2%)	29,525 (+2%)
note_history_successful_decrypting_time_in_ms	⚠️ 2,461 (-16%)	4,858 (-12%)
note_history_trial_decrypting_time_in_ms	79.4 (-18%)	233 (+60%)
node_database_size_in_bytes	3,619,123	3,721,014
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	211	44,709	25,561
private-kernel-ordering	138 (+1%)	30,953	14,809
base-rollup	1,079	128,374	881
root-rollup	82.9 (-2%)	4,088	889
private-kernel-inner	288	71,236	25,561
public-kernel-private-input	203	31,891	25,561
public-kernel-non-first-iteration	201 (+1%)	31,933	25,561
merge-rollup	9.80 (+1%)	2,608	881

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	8192 leaves
batch_insert_into_append_only_tree_ms	11.1	93.0 (+8%)	13.7 (-12%)	17.9	23.7 (+1%)	50.1 (+2%)	91.6 (+5%)	242 (+1%)	475 (+2%)	911 (+1%)	3,569 (+1%)
batch_insert_into_indexed_tree_ms	N/A	N/A	N/A	64.9 (+1%)	N/A	59.7	111 (+1%)	360 (+1%)	741 (+2%)	1,405 (+1%)	5,803 (+2%)

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts	1 deployed contracts
tx_size_in_bytes	15,487	31,877

Transaction processing duration by data writes.

Metric	0 new commitments	1 new commitments
tx_pxe_processing_time_ms	342 (-1%)	966

Metric	1 public data writes
tx_sequencer_processing_time_ms	532 (+1%)

yarn-project/pxe/src/contract_data_oracle/index.ts

just-mitch · 2024-01-08T17:21:34Z

yarn-project/pxe/src/database/kv_pxe_database.ts

+   * @param contractAddress - the contract address to remove deferred notes for
+   * @returns an array of the removed deferred notes
+   *
+   * @remarks We only remove indices from the deferred notes by contract map, but not the actual deferred notes.


Eventually I expect we will need to consolidate these delete vectors.

just-mitch · 2024-01-08T17:24:07Z

yarn-project/pxe/src/synchronizer/synchronizer.ts

@@ -17,9 +28,8 @@ import { NoteProcessor } from '../note_processor/index.js';
 * in sync with the blockchain while handling retries and errors gracefully.
 */
 export class Synchronizer {
-  private runningPromise?: Promise<void>;
+  private runningPromise?: RunningPromise;


unrelated, but quick cleanup of previous work.

Glad that has come in useful. I wrote it ages ago and I think it's in a few places.

yarn-project/pxe/src/synchronizer/synchronizer.ts

alexghr

alexghr · 2024-01-09T11:03:45Z

yarn-project/acir-simulator/src/client/db_oracle.ts

+export class ContractNotFoundError extends Error {
+  constructor(contractAddress: string) {
+    super(`DB has no contract with address ${contractAddress}`);
+  }
+}


This is great! We should create more custom error classes in the future. They should help us return better error messages through JSON-RPC too.

alexghr · 2024-01-09T11:10:16Z

yarn-project/pxe/src/contract_data_oracle/index.ts

+    const tree = await this.getTree(contractAddress);
+    return tree.contract.getFunctionArtifactByName(functionName);


Was this changed so it would warm up the this.trees cache for later use?
LE: saw the changes in ContractDao to clean up function lookups

alexghr · 2024-01-09T11:21:22Z

yarn-project/pxe/src/synchronizer/synchronizer.ts

+   * Retry decoding any deferred notes for the specified contract address.
+   * @param contractAddress - the contract address that has just been added
+   */
+  public async reprocessDeferredNotesForContract(contractAddress: AztecAddress) {


What do you think about running this function in the jobQueue so that it has a consistent view of nullifiers and db state? 🤔

Yes definitely an oversight!

alexghr · 2024-01-09T11:24:11Z

yarn-project/pxe/src/synchronizer/synchronizer.ts

+          relevantNullifiers.push(nullifier);
+        }
+      }
+      await this.db.removeNullifiedNotes(relevantNullifiers, publicKey);


Great catch!

I think it might be worth refactoring the database in the future to store nullifiers separately of notes so that we might run queries like where note.nullifier not in nullifiers

Interesting. You're suggesting we keep track of all seen nullifiers? That would remove the need for the findLeafIndex call, but the state bloat might concern me. Perhaps if the pattern we want to support is only note.nullifier not in nullifiers, we could get away with only storing a bloom filter locally, and fall back to findLeafIndex if the filter returns true.

Broadly though, I don't understand the benefit of the ability to run those kinds of queries if we're already pruning nullified notes.

PhilWindle · 2024-01-09T11:16:54Z

yarn-project/pxe/src/pxe_service/pxe_service.ts

@@ -205,9 +203,11 @@ export class PXEService implements PXE {
    const contractDaos = contracts.map(c => new ContractDao(c.artifact, c.completeAddress, c.portalContract));


I'm wondering if we should now enqueue the contents of this function using the job queue?

Yes for sure. Thanks!

Actually, as Alex mentioned below, I think it is fine to just synchronize the call to this.synchronizer.reprocessDeferredNotesForContract

PhilWindle · 2024-01-09T11:17:28Z

yarn-project/pxe/src/synchronizer/synchronizer.ts

@@ -17,9 +28,8 @@ import { NoteProcessor } from '../note_processor/index.js';
 * in sync with the blockchain while handling retries and errors gracefully.
 */
 export class Synchronizer {
-  private runningPromise?: Promise<void>;
+  private runningPromise?: RunningPromise;


Glad that has come in useful. I wrote it ages ago and I think it's in a few places.

compute nullifiers for deferred notes when decoded

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.18.0</summary> ## [0.18.0](aztec-packages-v0.17.0...aztec-packages-v0.18.0) (2024-01-16) ### ⚠ BREAKING CHANGES * Remove `Directive::Quotient` ([#4019](#4019)) * define key type in maps ([#3841](#3841)) * implement keccakf1600 in brillig ([#3914](#3914)) * add blake3 opcode to brillig ([#3913](#3913)) * Remove opcode supported from the backend ([#3889](#3889)) ### Features * `PartialStateReference` and `StateReference` structs ([#3827](#3827)) ([3ba0369](3ba0369)) * `StateDiffHints` ([#3919](#3919)) ([8774795](8774795)), closes [#3916](#3916) * Acir cleanup ([#3845](#3845)) ([390b84c](390b84c)) * Add ACIR opcodes for ECADD and ECDOUBLE ([#3878](#3878)) ([537630f](537630f)) * Add blake3 opcode to brillig ([#3913](#3913)) ([34fad0a](34fad0a)) * Add data availability oracle ([#3897](#3897)) ([5441753](5441753)), closes [#3890](#3890) * Add str support for args + add name/symbol/decimal to token ([#3862](#3862)) ([0bf5d8c](0bf5d8c)) * Bench bb in pr's, docker shell utils ([#3561](#3561)) ([5408919](5408919)) * Benchmark protogalaxy prover ([#3958](#3958)) ([5843722](5843722)) * Benchmarks for basic functionality and IPA improvements ([#4004](#4004)) ([fd1f619](fd1f619)) * Body hash as buffer in TS + minor cleanup ([#4012](#4012)) ([e28a6bf](e28a6bf)) * Bootstrap cache v2 ([#3876](#3876)) ([331598d](331598d)) * Counters in private functions ([#3850](#3850)) ([23bbf75](23bbf75)) * Generate json blocks for tests ([#3923](#3923)) ([a09fd2a](a09fd2a)) * Implement keccakf1600 in brillig ([#3914](#3914)) ([a182381](a182381)) * Parallel IPA ([#3882](#3882)) ([7002a33](7002a33)) * Pil lookups w/ xor table example ([#3880](#3880)) ([544d24e](544d24e)) * Poseidon2 stdlib impl ([#3551](#3551)) ([50b4a72](50b4a72)) * Protogalaxy Decider and complete folding tests ([#3657](#3657)) ([cfdaf9c](cfdaf9c)) * Reexport `protocol_types` from `aztec-nr` ([#3926](#3926)) ([9bd22f7](9bd22f7)) * Relations vs widgets benchmarking ([#3931](#3931)) ([3af64ef](3af64ef)) * Remove opcode supported from the backend ([#3889](#3889)) ([1fd135c](1fd135c)) * Reorganize acir composer ([#3957](#3957)) ([e6232e8](e6232e8)) * Replace bitwise ORs in `U256:from_bytes32` with addition ([#3947](#3947)) ([efd7660](efd7660)) * Standalone calldata test ([#3842](#3842)) ([7353a35](7353a35)) * Sync with main noir repo ([#3939](#3939)) ([69c7e99](69c7e99)) * Update noir ([#3979](#3979)) ([271de71](271de71)) * Verify state hash is correct before publishing to L1 ([#3915](#3915)) ([a53c261](a53c261)) ### Bug Fixes * **aztec-nr:** Broken nargo url ([#3925](#3925)) ([034bc30](034bc30)) * Back out the buggy branch condition ([#3988](#3988)) ([9f61ed1](9f61ed1)) * Bb unnecessary env var ([#3901](#3901)) ([f127e5a](f127e5a)) * Broken links by paterson1 ([#3902](#3902)) ([6783aaa](6783aaa)) * **cli:** Unbox command should run as release ([#3974](#3974)) ([80c3805](80c3805)) * **cli:** Unbox should set up the accounts package ([#3972](#3972)) ([065e988](065e988)) * Docusaurus start command ([#3968](#3968)) ([87c0b07](87c0b07)) * Dont spam logs with yarn install ([#4027](#4027)) ([949c5ab](949c5ab)) * Fix compilation of `token` box ([#3981](#3981)) ([0f994d0](0f994d0)) * Mirror_noir_subrepo.yml erroring ([#3954](#3954)) ([2ac1b9c](2ac1b9c)) * Quote the glob string for ignore branches for protocol-circuits-gate-diff ([#3990](#3990)) ([a43889d](a43889d)) * Reprocess notes in pxe when a new contract is added ([#3867](#3867)) ([ccbff99](ccbff99)) * Segment tree insertion stats by depth ([#4029](#4029)) ([2787bae](2787bae)) * Store blockhash alongside blocks ([#3950](#3950)) ([12b07fa](12b07fa)), closes [#3870](#3870) * Swap branch exclusion for release-please on report gates diff workflow ([#3994](#3994)) ([70b2ffd](70b2ffd)) * Typos in authwit.md by czepluch ([#3921](#3921)) ([4b9d0f4](4b9d0f4)) * Yellowpaper docusaurus start command ([#3969](#3969)) ([4977cbc](4977cbc)) ### Miscellaneous * Add a link back to `aztec-packages` in noir sync PR ([#4018](#4018)) ([7d89f3b](7d89f3b)) * Catch up note processors could be synced more efficiently ([#3933](#3933)) ([df54f33](df54f33)) * **ci:** Require boxes CI to succeed ([#3983](#3983)) ([93cbea1](93cbea1)) * Cleanup duplicated methods for structs after traits ([#3912](#3912)) ([60b59da](60b59da)) * Cleanup sandbox dependent tests ([#3861](#3861)) ([158c5be](158c5be)) * Codegen acir opcodes after renaming arithmetic to assertzero ([#3896](#3896)) ([c710ce1](c710ce1)) * Define key type in maps ([#3841](#3841)) ([cf15adb](cf15adb)) * Delete the compiler from `noir-compiler` ([#3959](#3959)) ([9aa0986](9aa0986)) * Deploy canary release of bb.js to npm via CCI ([#3917](#3917)) ([bdeb10c](bdeb10c)) * Do not fail jq dependencies check for package with no deps ([#3894](#3894)) ([22c65bf](22c65bf)) * Do not pass redundant txNullifier when computing notes ([#3943](#3943)) ([9355cda](9355cda)) * Do not run CCI on release-please branches ([#3984](#3984)) ([c38dbd2](c38dbd2)) * **docs:** Fix ts code in token bridge tutorial ([#3888](#3888)) ([f53f8ed](f53f8ed)) * **docs:** Move map keys to new release in migration guide ([#3977](#3977)) ([6356d94](6356d94)) * **docs:** Update missing @aztec/accounts missing import in testing.md ([#3903](#3903)) ([755668a](755668a)) * Document `witness_buf_to_witness_data` ([#3940](#3940)) ([fbaa726](fbaa726)) * End to end test node & pxe persistence ([#3911](#3911)) ([6164ccd](6164ccd)) * Enforce immutable yarn installs in CI ([#3964](#3964)) ([f3104ac](f3104ac)) * Fix rust tests ([#3963](#3963)) ([a907c3b](a907c3b)) * Fix sidebar index links ([#3942](#3942)) ([984f1f8](984f1f8)) * Git subrepo commit (merge) noir ([#3955](#3955)) ([2c2bc69](2c2bc69)) * Introduce EventSelector class ([#3960](#3960)) ([7315f2c](7315f2c)) * Investigate P2P test ([#3929](#3929)) ([0fca2c4](0fca2c4)) * Move types to circuit-types ([#3967](#3967)) ([f81b7c0](f81b7c0)) * Persistence uses TokenContract ([#3930](#3930)) ([1a052c4](1a052c4)) * Pull in noir from upstream ([#3904](#3904)) ([ab07e7e](ab07e7e)) * Recreated types package without circuits.js dependency ([#3970](#3970)) ([fc1d539](fc1d539)) * Refactor serialisation functions and sibling paths ([#3980](#3980)) ([1a936fc](1a936fc)) * Remove 'extern template's, expand macros ([#3953](#3953)) ([5fe9908](5fe9908)) * Remove `Directive::Quotient` ([#4019](#4019)) ([824d76f](824d76f)) * Remove sandbox and cli npm pkgs ([#3567](#3567)) ([a8cf1bf](a8cf1bf)) * Reorganize benchmarks ([#3909](#3909)) ([730766b](730766b)) * Replace `AztecU128` with `U128` ([#3951](#3951)) ([e3b288d](e3b288d)) * Replace relative paths to noir-protocol-circuits ([2c25f04](2c25f04)) * Replace relative paths to noir-protocol-circuits ([0c8b770](0c8b770)) * Replace relative paths to noir-protocol-circuits ([d3819ba](d3819ba)) * Standardise toml parsers ([#3910](#3910)) ([963035f](963035f)) * Sync noir ([#4025](#4025)) ([4e90d7b](4e90d7b)) * Sync Noir repo ([#4020](#4020)) ([876603e](876603e)) * **yellowpaper:** Cleanup avm sidebar, fix filename case ([#3952](#3952)) ([5211060](5211060)) * Yp docs sidebar (with some auto-formatting) ([#3893](#3893)) ([f7b007a](f7b007a)) ### Documentation * Noir git subrepo usage ([#3962](#3962)) ([2e4c9de](2e4c9de)) * Streamlined pr template ([#3932](#3932)) ([5ec1559](5ec1559)) * **yellowpaper:** Avm call pointers, bytecode lookups, circuit io ([#3898](#3898)) ([45e1ed2](45e1ed2)) * **yellowpaper:** Avm circuit architecture ([#3934](#3934)) ([6aed1d0](6aed1d0)) * **yellowpaper:** First draft of avm circuit memory ([#3865](#3865)) ([f689297](f689297)) * **yellowpaper:** Logs ([#4016](#4016)) ([d734c79](d734c79)) * **yellowpaper:** Update AVM spec for with "daGasLeft", some cleanup ([#3956](#3956)) ([a9537fb](a9537fb)) </details> <details><summary>barretenberg.js: 0.18.0</summary> ## [0.18.0](barretenberg.js-v0.17.0...barretenberg.js-v0.18.0) (2024-01-16) ### Features * Bootstrap cache v2 ([#3876](#3876)) ([331598d](331598d)) ### Bug Fixes * Dont spam logs with yarn install ([#4027](#4027)) ([949c5ab](949c5ab)) </details> <details><summary>barretenberg: 0.18.0</summary> ## [0.18.0](barretenberg-v0.17.0...barretenberg-v0.18.0) (2024-01-16) ### ⚠ BREAKING CHANGES * Remove `Directive::Quotient` ([#4019](#4019)) * implement keccakf1600 in brillig ([#3914](#3914)) * add blake3 opcode to brillig ([#3913](#3913)) * Remove opcode supported from the backend ([#3889](#3889)) ### Features * Acir cleanup ([#3845](#3845)) ([390b84c](390b84c)) * Add ACIR opcodes for ECADD and ECDOUBLE ([#3878](#3878)) ([537630f](537630f)) * Add blake3 opcode to brillig ([#3913](#3913)) ([34fad0a](34fad0a)) * Bench bb in pr's, docker shell utils ([#3561](#3561)) ([5408919](5408919)) * Benchmark protogalaxy prover ([#3958](#3958)) ([5843722](5843722)) * Benchmarks for basic functionality and IPA improvements ([#4004](#4004)) ([fd1f619](fd1f619)) * Bootstrap cache v2 ([#3876](#3876)) ([331598d](331598d)) * Implement keccakf1600 in brillig ([#3914](#3914)) ([a182381](a182381)) * Parallel IPA ([#3882](#3882)) ([7002a33](7002a33)) * Pil lookups w/ xor table example ([#3880](#3880)) ([544d24e](544d24e)) * Poseidon2 stdlib impl ([#3551](#3551)) ([50b4a72](50b4a72)) * Protogalaxy Decider and complete folding tests ([#3657](#3657)) ([cfdaf9c](cfdaf9c)) * Relations vs widgets benchmarking ([#3931](#3931)) ([3af64ef](3af64ef)) * Remove opcode supported from the backend ([#3889](#3889)) ([1fd135c](1fd135c)) * Reorganize acir composer ([#3957](#3957)) ([e6232e8](e6232e8)) * Standalone calldata test ([#3842](#3842)) ([7353a35](7353a35)) ### Bug Fixes * Bb unnecessary env var ([#3901](#3901)) ([f127e5a](f127e5a)) ### Miscellaneous * Codegen acir opcodes after renaming arithmetic to assertzero ([#3896](#3896)) ([c710ce1](c710ce1)) * Document `witness_buf_to_witness_data` ([#3940](#3940)) ([fbaa726](fbaa726)) * Remove 'extern template's, expand macros ([#3953](#3953)) ([5fe9908](5fe9908)) * Remove `Directive::Quotient` ([#4019](#4019)) ([824d76f](824d76f)) * Reorganize benchmarks ([#3909](#3909)) ([730766b](730766b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-packages: 0.18.0</summary> ## [0.18.0](AztecProtocol/aztec-packages@aztec-packages-v0.17.0...aztec-packages-v0.18.0) (2024-01-16) ### ⚠ BREAKING CHANGES * Remove `Directive::Quotient` ([#4019](AztecProtocol/aztec-packages#4019)) * define key type in maps ([#3841](AztecProtocol/aztec-packages#3841)) * implement keccakf1600 in brillig ([#3914](AztecProtocol/aztec-packages#3914)) * add blake3 opcode to brillig ([#3913](AztecProtocol/aztec-packages#3913)) * Remove opcode supported from the backend ([#3889](AztecProtocol/aztec-packages#3889)) ### Features * `PartialStateReference` and `StateReference` structs ([#3827](AztecProtocol/aztec-packages#3827)) ([3ba0369](AztecProtocol/aztec-packages@3ba0369)) * `StateDiffHints` ([#3919](AztecProtocol/aztec-packages#3919)) ([8774795](AztecProtocol/aztec-packages@8774795)), closes [#3916](AztecProtocol/aztec-packages#3916) * Acir cleanup ([#3845](AztecProtocol/aztec-packages#3845)) ([390b84c](AztecProtocol/aztec-packages@390b84c)) * Add ACIR opcodes for ECADD and ECDOUBLE ([#3878](AztecProtocol/aztec-packages#3878)) ([537630f](AztecProtocol/aztec-packages@537630f)) * Add blake3 opcode to brillig ([#3913](AztecProtocol/aztec-packages#3913)) ([34fad0a](AztecProtocol/aztec-packages@34fad0a)) * Add data availability oracle ([#3897](AztecProtocol/aztec-packages#3897)) ([5441753](AztecProtocol/aztec-packages@5441753)), closes [#3890](AztecProtocol/aztec-packages#3890) * Add str support for args + add name/symbol/decimal to token ([#3862](AztecProtocol/aztec-packages#3862)) ([0bf5d8c](AztecProtocol/aztec-packages@0bf5d8c)) * Bench bb in pr's, docker shell utils ([#3561](AztecProtocol/aztec-packages#3561)) ([5408919](AztecProtocol/aztec-packages@5408919)) * Benchmark protogalaxy prover ([#3958](AztecProtocol/aztec-packages#3958)) ([5843722](AztecProtocol/aztec-packages@5843722)) * Benchmarks for basic functionality and IPA improvements ([#4004](AztecProtocol/aztec-packages#4004)) ([fd1f619](AztecProtocol/aztec-packages@fd1f619)) * Body hash as buffer in TS + minor cleanup ([#4012](AztecProtocol/aztec-packages#4012)) ([e28a6bf](AztecProtocol/aztec-packages@e28a6bf)) * Bootstrap cache v2 ([#3876](AztecProtocol/aztec-packages#3876)) ([331598d](AztecProtocol/aztec-packages@331598d)) * Counters in private functions ([#3850](AztecProtocol/aztec-packages#3850)) ([23bbf75](AztecProtocol/aztec-packages@23bbf75)) * Generate json blocks for tests ([#3923](AztecProtocol/aztec-packages#3923)) ([a09fd2a](AztecProtocol/aztec-packages@a09fd2a)) * Implement keccakf1600 in brillig ([#3914](AztecProtocol/aztec-packages#3914)) ([a182381](AztecProtocol/aztec-packages@a182381)) * Parallel IPA ([#3882](AztecProtocol/aztec-packages#3882)) ([7002a33](AztecProtocol/aztec-packages@7002a33)) * Pil lookups w/ xor table example ([#3880](AztecProtocol/aztec-packages#3880)) ([544d24e](AztecProtocol/aztec-packages@544d24e)) * Poseidon2 stdlib impl ([#3551](AztecProtocol/aztec-packages#3551)) ([50b4a72](AztecProtocol/aztec-packages@50b4a72)) * Protogalaxy Decider and complete folding tests ([#3657](AztecProtocol/aztec-packages#3657)) ([cfdaf9c](AztecProtocol/aztec-packages@cfdaf9c)) * Reexport `protocol_types` from `aztec-nr` ([#3926](AztecProtocol/aztec-packages#3926)) ([9bd22f7](AztecProtocol/aztec-packages@9bd22f7)) * Relations vs widgets benchmarking ([#3931](AztecProtocol/aztec-packages#3931)) ([3af64ef](AztecProtocol/aztec-packages@3af64ef)) * Remove opcode supported from the backend ([#3889](AztecProtocol/aztec-packages#3889)) ([1fd135c](AztecProtocol/aztec-packages@1fd135c)) * Reorganize acir composer ([#3957](AztecProtocol/aztec-packages#3957)) ([e6232e8](AztecProtocol/aztec-packages@e6232e8)) * Replace bitwise ORs in `U256:from_bytes32` with addition ([#3947](AztecProtocol/aztec-packages#3947)) ([efd7660](AztecProtocol/aztec-packages@efd7660)) * Standalone calldata test ([#3842](AztecProtocol/aztec-packages#3842)) ([7353a35](AztecProtocol/aztec-packages@7353a35)) * Sync with main noir repo ([#3939](AztecProtocol/aztec-packages#3939)) ([69c7e99](AztecProtocol/aztec-packages@69c7e99)) * Update noir ([#3979](AztecProtocol/aztec-packages#3979)) ([271de71](AztecProtocol/aztec-packages@271de71)) * Verify state hash is correct before publishing to L1 ([#3915](AztecProtocol/aztec-packages#3915)) ([a53c261](AztecProtocol/aztec-packages@a53c261)) ### Bug Fixes * **aztec-nr:** Broken nargo url ([#3925](AztecProtocol/aztec-packages#3925)) ([034bc30](AztecProtocol/aztec-packages@034bc30)) * Back out the buggy branch condition ([#3988](AztecProtocol/aztec-packages#3988)) ([9f61ed1](AztecProtocol/aztec-packages@9f61ed1)) * Bb unnecessary env var ([#3901](AztecProtocol/aztec-packages#3901)) ([f127e5a](AztecProtocol/aztec-packages@f127e5a)) * Broken links by paterson1 ([#3902](AztecProtocol/aztec-packages#3902)) ([6783aaa](AztecProtocol/aztec-packages@6783aaa)) * **cli:** Unbox command should run as release ([#3974](AztecProtocol/aztec-packages#3974)) ([80c3805](AztecProtocol/aztec-packages@80c3805)) * **cli:** Unbox should set up the accounts package ([#3972](AztecProtocol/aztec-packages#3972)) ([065e988](AztecProtocol/aztec-packages@065e988)) * Docusaurus start command ([#3968](AztecProtocol/aztec-packages#3968)) ([87c0b07](AztecProtocol/aztec-packages@87c0b07)) * Dont spam logs with yarn install ([#4027](AztecProtocol/aztec-packages#4027)) ([949c5ab](AztecProtocol/aztec-packages@949c5ab)) * Fix compilation of `token` box ([#3981](AztecProtocol/aztec-packages#3981)) ([0f994d0](AztecProtocol/aztec-packages@0f994d0)) * Mirror_noir_subrepo.yml erroring ([#3954](AztecProtocol/aztec-packages#3954)) ([2ac1b9c](AztecProtocol/aztec-packages@2ac1b9c)) * Quote the glob string for ignore branches for protocol-circuits-gate-diff ([#3990](AztecProtocol/aztec-packages#3990)) ([a43889d](AztecProtocol/aztec-packages@a43889d)) * Reprocess notes in pxe when a new contract is added ([#3867](AztecProtocol/aztec-packages#3867)) ([ccbff99](AztecProtocol/aztec-packages@ccbff99)) * Segment tree insertion stats by depth ([#4029](AztecProtocol/aztec-packages#4029)) ([2787bae](AztecProtocol/aztec-packages@2787bae)) * Store blockhash alongside blocks ([#3950](AztecProtocol/aztec-packages#3950)) ([12b07fa](AztecProtocol/aztec-packages@12b07fa)), closes [#3870](AztecProtocol/aztec-packages#3870) * Swap branch exclusion for release-please on report gates diff workflow ([#3994](AztecProtocol/aztec-packages#3994)) ([70b2ffd](AztecProtocol/aztec-packages@70b2ffd)) * Typos in authwit.md by czepluch ([#3921](AztecProtocol/aztec-packages#3921)) ([4b9d0f4](AztecProtocol/aztec-packages@4b9d0f4)) * Yellowpaper docusaurus start command ([#3969](AztecProtocol/aztec-packages#3969)) ([4977cbc](AztecProtocol/aztec-packages@4977cbc)) ### Miscellaneous * Add a link back to `aztec-packages` in noir sync PR ([#4018](AztecProtocol/aztec-packages#4018)) ([7d89f3b](AztecProtocol/aztec-packages@7d89f3b)) * Catch up note processors could be synced more efficiently ([#3933](AztecProtocol/aztec-packages#3933)) ([df54f33](AztecProtocol/aztec-packages@df54f33)) * **ci:** Require boxes CI to succeed ([#3983](AztecProtocol/aztec-packages#3983)) ([93cbea1](AztecProtocol/aztec-packages@93cbea1)) * Cleanup duplicated methods for structs after traits ([#3912](AztecProtocol/aztec-packages#3912)) ([60b59da](AztecProtocol/aztec-packages@60b59da)) * Cleanup sandbox dependent tests ([#3861](AztecProtocol/aztec-packages#3861)) ([158c5be](AztecProtocol/aztec-packages@158c5be)) * Codegen acir opcodes after renaming arithmetic to assertzero ([#3896](AztecProtocol/aztec-packages#3896)) ([c710ce1](AztecProtocol/aztec-packages@c710ce1)) * Define key type in maps ([#3841](AztecProtocol/aztec-packages#3841)) ([cf15adb](AztecProtocol/aztec-packages@cf15adb)) * Delete the compiler from `noir-compiler` ([#3959](AztecProtocol/aztec-packages#3959)) ([9aa0986](AztecProtocol/aztec-packages@9aa0986)) * Deploy canary release of bb.js to npm via CCI ([#3917](AztecProtocol/aztec-packages#3917)) ([bdeb10c](AztecProtocol/aztec-packages@bdeb10c)) * Do not fail jq dependencies check for package with no deps ([#3894](AztecProtocol/aztec-packages#3894)) ([22c65bf](AztecProtocol/aztec-packages@22c65bf)) * Do not pass redundant txNullifier when computing notes ([#3943](AztecProtocol/aztec-packages#3943)) ([9355cda](AztecProtocol/aztec-packages@9355cda)) * Do not run CCI on release-please branches ([#3984](AztecProtocol/aztec-packages#3984)) ([c38dbd2](AztecProtocol/aztec-packages@c38dbd2)) * **docs:** Fix ts code in token bridge tutorial ([#3888](AztecProtocol/aztec-packages#3888)) ([f53f8ed](AztecProtocol/aztec-packages@f53f8ed)) * **docs:** Move map keys to new release in migration guide ([#3977](AztecProtocol/aztec-packages#3977)) ([6356d94](AztecProtocol/aztec-packages@6356d94)) * **docs:** Update missing @aztec/accounts missing import in testing.md ([#3903](AztecProtocol/aztec-packages#3903)) ([755668a](AztecProtocol/aztec-packages@755668a)) * Document `witness_buf_to_witness_data` ([#3940](AztecProtocol/aztec-packages#3940)) ([fbaa726](AztecProtocol/aztec-packages@fbaa726)) * End to end test node & pxe persistence ([#3911](AztecProtocol/aztec-packages#3911)) ([6164ccd](AztecProtocol/aztec-packages@6164ccd)) * Enforce immutable yarn installs in CI ([#3964](AztecProtocol/aztec-packages#3964)) ([f3104ac](AztecProtocol/aztec-packages@f3104ac)) * Fix rust tests ([#3963](AztecProtocol/aztec-packages#3963)) ([a907c3b](AztecProtocol/aztec-packages@a907c3b)) * Fix sidebar index links ([#3942](AztecProtocol/aztec-packages#3942)) ([984f1f8](AztecProtocol/aztec-packages@984f1f8)) * Git subrepo commit (merge) noir ([#3955](AztecProtocol/aztec-packages#3955)) ([2c2bc69](AztecProtocol/aztec-packages@2c2bc69)) * Introduce EventSelector class ([#3960](AztecProtocol/aztec-packages#3960)) ([7315f2c](AztecProtocol/aztec-packages@7315f2c)) * Investigate P2P test ([#3929](AztecProtocol/aztec-packages#3929)) ([0fca2c4](AztecProtocol/aztec-packages@0fca2c4)) * Move types to circuit-types ([#3967](AztecProtocol/aztec-packages#3967)) ([f81b7c0](AztecProtocol/aztec-packages@f81b7c0)) * Persistence uses TokenContract ([#3930](AztecProtocol/aztec-packages#3930)) ([1a052c4](AztecProtocol/aztec-packages@1a052c4)) * Pull in noir from upstream ([#3904](AztecProtocol/aztec-packages#3904)) ([ab07e7e](AztecProtocol/aztec-packages@ab07e7e)) * Recreated types package without circuits.js dependency ([#3970](AztecProtocol/aztec-packages#3970)) ([fc1d539](AztecProtocol/aztec-packages@fc1d539)) * Refactor serialisation functions and sibling paths ([#3980](AztecProtocol/aztec-packages#3980)) ([1a936fc](AztecProtocol/aztec-packages@1a936fc)) * Remove 'extern template's, expand macros ([#3953](AztecProtocol/aztec-packages#3953)) ([5fe9908](AztecProtocol/aztec-packages@5fe9908)) * Remove `Directive::Quotient` ([#4019](AztecProtocol/aztec-packages#4019)) ([824d76f](AztecProtocol/aztec-packages@824d76f)) * Remove sandbox and cli npm pkgs ([#3567](AztecProtocol/aztec-packages#3567)) ([a8cf1bf](AztecProtocol/aztec-packages@a8cf1bf)) * Reorganize benchmarks ([#3909](AztecProtocol/aztec-packages#3909)) ([730766b](AztecProtocol/aztec-packages@730766b)) * Replace `AztecU128` with `U128` ([#3951](AztecProtocol/aztec-packages#3951)) ([e3b288d](AztecProtocol/aztec-packages@e3b288d)) * Replace relative paths to noir-protocol-circuits ([2c25f04](AztecProtocol/aztec-packages@2c25f04)) * Replace relative paths to noir-protocol-circuits ([0c8b770](AztecProtocol/aztec-packages@0c8b770)) * Replace relative paths to noir-protocol-circuits ([d3819ba](AztecProtocol/aztec-packages@d3819ba)) * Standardise toml parsers ([#3910](AztecProtocol/aztec-packages#3910)) ([963035f](AztecProtocol/aztec-packages@963035f)) * Sync noir ([#4025](AztecProtocol/aztec-packages#4025)) ([4e90d7b](AztecProtocol/aztec-packages@4e90d7b)) * Sync Noir repo ([#4020](AztecProtocol/aztec-packages#4020)) ([876603e](AztecProtocol/aztec-packages@876603e)) * **yellowpaper:** Cleanup avm sidebar, fix filename case ([#3952](AztecProtocol/aztec-packages#3952)) ([5211060](AztecProtocol/aztec-packages@5211060)) * Yp docs sidebar (with some auto-formatting) ([#3893](AztecProtocol/aztec-packages#3893)) ([f7b007a](AztecProtocol/aztec-packages@f7b007a)) ### Documentation * Noir git subrepo usage ([#3962](AztecProtocol/aztec-packages#3962)) ([2e4c9de](AztecProtocol/aztec-packages@2e4c9de)) * Streamlined pr template ([#3932](AztecProtocol/aztec-packages#3932)) ([5ec1559](AztecProtocol/aztec-packages@5ec1559)) * **yellowpaper:** Avm call pointers, bytecode lookups, circuit io ([#3898](AztecProtocol/aztec-packages#3898)) ([45e1ed2](AztecProtocol/aztec-packages@45e1ed2)) * **yellowpaper:** Avm circuit architecture ([#3934](AztecProtocol/aztec-packages#3934)) ([6aed1d0](AztecProtocol/aztec-packages@6aed1d0)) * **yellowpaper:** First draft of avm circuit memory ([#3865](AztecProtocol/aztec-packages#3865)) ([f689297](AztecProtocol/aztec-packages@f689297)) * **yellowpaper:** Logs ([#4016](AztecProtocol/aztec-packages#4016)) ([d734c79](AztecProtocol/aztec-packages@d734c79)) * **yellowpaper:** Update AVM spec for with "daGasLeft", some cleanup ([#3956](AztecProtocol/aztec-packages#3956)) ([a9537fb](AztecProtocol/aztec-packages@a9537fb)) </details> <details><summary>barretenberg.js: 0.18.0</summary> ## [0.18.0](AztecProtocol/aztec-packages@barretenberg.js-v0.17.0...barretenberg.js-v0.18.0) (2024-01-16) ### Features * Bootstrap cache v2 ([#3876](AztecProtocol/aztec-packages#3876)) ([331598d](AztecProtocol/aztec-packages@331598d)) ### Bug Fixes * Dont spam logs with yarn install ([#4027](AztecProtocol/aztec-packages#4027)) ([949c5ab](AztecProtocol/aztec-packages@949c5ab)) </details> <details><summary>barretenberg: 0.18.0</summary> ## [0.18.0](AztecProtocol/aztec-packages@barretenberg-v0.17.0...barretenberg-v0.18.0) (2024-01-16) ### ⚠ BREAKING CHANGES * Remove `Directive::Quotient` ([#4019](AztecProtocol/aztec-packages#4019)) * implement keccakf1600 in brillig ([#3914](AztecProtocol/aztec-packages#3914)) * add blake3 opcode to brillig ([#3913](AztecProtocol/aztec-packages#3913)) * Remove opcode supported from the backend ([#3889](AztecProtocol/aztec-packages#3889)) ### Features * Acir cleanup ([#3845](AztecProtocol/aztec-packages#3845)) ([390b84c](AztecProtocol/aztec-packages@390b84c)) * Add ACIR opcodes for ECADD and ECDOUBLE ([#3878](AztecProtocol/aztec-packages#3878)) ([537630f](AztecProtocol/aztec-packages@537630f)) * Add blake3 opcode to brillig ([#3913](AztecProtocol/aztec-packages#3913)) ([34fad0a](AztecProtocol/aztec-packages@34fad0a)) * Bench bb in pr's, docker shell utils ([#3561](AztecProtocol/aztec-packages#3561)) ([5408919](AztecProtocol/aztec-packages@5408919)) * Benchmark protogalaxy prover ([#3958](AztecProtocol/aztec-packages#3958)) ([5843722](AztecProtocol/aztec-packages@5843722)) * Benchmarks for basic functionality and IPA improvements ([#4004](AztecProtocol/aztec-packages#4004)) ([fd1f619](AztecProtocol/aztec-packages@fd1f619)) * Bootstrap cache v2 ([#3876](AztecProtocol/aztec-packages#3876)) ([331598d](AztecProtocol/aztec-packages@331598d)) * Implement keccakf1600 in brillig ([#3914](AztecProtocol/aztec-packages#3914)) ([a182381](AztecProtocol/aztec-packages@a182381)) * Parallel IPA ([#3882](AztecProtocol/aztec-packages#3882)) ([7002a33](AztecProtocol/aztec-packages@7002a33)) * Pil lookups w/ xor table example ([#3880](AztecProtocol/aztec-packages#3880)) ([544d24e](AztecProtocol/aztec-packages@544d24e)) * Poseidon2 stdlib impl ([#3551](AztecProtocol/aztec-packages#3551)) ([50b4a72](AztecProtocol/aztec-packages@50b4a72)) * Protogalaxy Decider and complete folding tests ([#3657](AztecProtocol/aztec-packages#3657)) ([cfdaf9c](AztecProtocol/aztec-packages@cfdaf9c)) * Relations vs widgets benchmarking ([#3931](AztecProtocol/aztec-packages#3931)) ([3af64ef](AztecProtocol/aztec-packages@3af64ef)) * Remove opcode supported from the backend ([#3889](AztecProtocol/aztec-packages#3889)) ([1fd135c](AztecProtocol/aztec-packages@1fd135c)) * Reorganize acir composer ([#3957](AztecProtocol/aztec-packages#3957)) ([e6232e8](AztecProtocol/aztec-packages@e6232e8)) * Standalone calldata test ([#3842](AztecProtocol/aztec-packages#3842)) ([7353a35](AztecProtocol/aztec-packages@7353a35)) ### Bug Fixes * Bb unnecessary env var ([#3901](AztecProtocol/aztec-packages#3901)) ([f127e5a](AztecProtocol/aztec-packages@f127e5a)) ### Miscellaneous * Codegen acir opcodes after renaming arithmetic to assertzero ([#3896](AztecProtocol/aztec-packages#3896)) ([c710ce1](AztecProtocol/aztec-packages@c710ce1)) * Document `witness_buf_to_witness_data` ([#3940](AztecProtocol/aztec-packages#3940)) ([fbaa726](AztecProtocol/aztec-packages@fbaa726)) * Remove 'extern template's, expand macros ([#3953](AztecProtocol/aztec-packages#3953)) ([5fe9908](AztecProtocol/aztec-packages@5fe9908)) * Remove `Directive::Quotient` ([#4019](AztecProtocol/aztec-packages#4019)) ([824d76f](AztecProtocol/aztec-packages@824d76f)) * Reorganize benchmarks ([#3909](AztecProtocol/aztec-packages#3909)) ([730766b](AztecProtocol/aztec-packages@730766b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

…ol#3867) Add notion of a Deferred Note: a note that we can *decrypt*, but not decode because we don't have the contract artifact in our PXE database. We hang on to these deferred notes in a new store in our PXE database, and reprocess them when their contract is made available. When we reprocess and successfully decode, we also need to search if we have already nullified the newly decoded note.

just-mitch linked an issue Jan 7, 2024 that may be closed by this pull request

Reprocess notes in pxe when a new contract is added #2709

Closed

just-mitch changed the title ~~WIP: 2709 reprocess notes in pxe when a new contract is added~~ fix: 2709 reprocess notes in pxe when a new contract is added Jan 8, 2024

just-mitch requested review from alexghr and PhilWindle January 8, 2024 17:11

just-mitch commented Jan 8, 2024

View reviewed changes

yarn-project/pxe/src/contract_data_oracle/index.ts Outdated Show resolved Hide resolved

just-mitch commented Jan 8, 2024

View reviewed changes

yarn-project/pxe/src/synchronizer/synchronizer.ts Outdated Show resolved Hide resolved

just-mitch marked this pull request as ready for review January 8, 2024 18:06

alexghr approved these changes Jan 9, 2024

View reviewed changes

PhilWindle requested changes Jan 9, 2024

View reviewed changes

just-mitch force-pushed the 2709-reprocess-notes-in-pxe-when-a-new-contract-is-added branch from f1d2d78 to aa2613c Compare January 9, 2024 13:58

just-mitch requested a review from PhilWindle January 9, 2024 15:34

PhilWindle approved these changes Jan 9, 2024

View reviewed changes

just-mitch enabled auto-merge (squash) January 9, 2024 19:33

Mitchell Tracy added 8 commits January 9, 2024 14:33

utilize the RunningPromise for managing synchronizer loop

742f113

add test demonstrating the bug

8cd9031

detect the case where the contract does not exist

f1bccd5

store deferred notes in the pxe database

606863b

simple test passes

164fe25

remove deferred notes from pxe db

227bc11

compute nullifiers for deferred notes when decoded

cleanup comments, implementation for in-memory database

d1f7267

address PR feedback

0c18d84

just-mitch force-pushed the 2709-reprocess-notes-in-pxe-when-a-new-contract-is-added branch from aa2613c to 0c18d84 Compare January 9, 2024 19:33

start the pxe job queue in the constructor

3dd9535

just-mitch changed the title ~~fix: 2709 reprocess notes in pxe when a new contract is added~~ fix: reprocess notes in pxe when a new contract is added Jan 9, 2024

just-mitch merged commit ccbff99 into master Jan 9, 2024
78 of 81 checks passed

just-mitch deleted the 2709-reprocess-notes-in-pxe-when-a-new-contract-is-added branch January 9, 2024 20:37

AztecBot mentioned this pull request Jan 9, 2024

chore(master): Release 0.18.0 #3892

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: reprocess notes in pxe when a new contract is added #3867

fix: reprocess notes in pxe when a new contract is added #3867

just-mitch commented Jan 7, 2024 •

edited

Loading

AztecBot commented Jan 7, 2024 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

just-mitch Jan 8, 2024

just-mitch Jan 8, 2024

PhilWindle Jan 9, 2024

alexghr left a comment

alexghr Jan 9, 2024

alexghr Jan 9, 2024 •

edited

Loading

alexghr Jan 9, 2024 •

edited

Loading

just-mitch Jan 9, 2024 •

edited

Loading

alexghr Jan 9, 2024

just-mitch Jan 9, 2024

PhilWindle Jan 9, 2024

just-mitch Jan 9, 2024

just-mitch Jan 9, 2024

PhilWindle Jan 9, 2024

		const tree = await this.getTree(contractAddress);
		return tree.contract.getFunctionArtifactByName(functionName);

		@@ -205,9 +203,11 @@ export class PXEService implements PXE {
		const contractDaos = contracts.map(c => new ContractDao(c.artifact, c.completeAddress, c.portalContract));

fix: reprocess notes in pxe when a new contract is added #3867

fix: reprocess notes in pxe when a new contract is added #3867

Conversation

just-mitch commented Jan 7, 2024 • edited Loading

AztecBot commented Jan 7, 2024 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexghr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexghr Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

alexghr Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

just-mitch Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

just-mitch commented Jan 7, 2024 •

edited

Loading

AztecBot commented Jan 7, 2024 •

edited

Loading

alexghr Jan 9, 2024 •

edited

Loading

alexghr Jan 9, 2024 •

edited

Loading

just-mitch Jan 9, 2024 •

edited

Loading