Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add persistent archiver store #3361

Closed
1 task done
Tracked by #3366
alexghr opened this issue Nov 20, 2023 · 0 comments · Fixed by #3410 or #3435
Closed
1 task done
Tracked by #3366

Add persistent archiver store #3361

alexghr opened this issue Nov 20, 2023 · 0 comments · Fixed by #3410 or #3435
Assignees
@alexghr

Comments

@alexghr
Copy link
Contributor

alexghr commented Nov 20, 2023 •
edited
Loading

The Archiver is responsible for pulling processed L2 blocks from calldata on L1. Its current store keeps the data in memory which is fast but whenever the node is restarted it has to sync from the beginning.

Add a new implementation alongside the MemoryArchiverStore that's backed by a persistent data store.

Requirements

The archiver store will see a mix of read and write workload with a bias towards reads:

  • reads should be consistent and it should allow for multiple simultaneous reads
  • writes should be atomic. During block processing a lot of data will be written to the store and we need to make sure partial data is not read

With this persistent store the archiver on boot should continue synchronizing from where it left off. To ensure consistency the archiver will have to validate the stored state with the state on the chain.

If the rollup contract address on L1 changes then the archiver will have to clear the store and start over from scratch.

Questions

  • Would checking the most recently committed block be enough?
    We'd have to check all blocks to be confident the data on disk is valid. It's not worth it as it would be equivalent to syncing from 0 again. If data is corrupted things downstream of the Archiver will start failing.

Database choice

LMDB was chosen for its good Node.js bindings, fast reads and ACID compliance.
@alexghr alexghr self-assigned this Nov 20, 2023
@alexghr alexghr added this to A3 Nov 20, 2023
@github-project-automation github-project-automation bot moved this to Todo in A3 Nov 20, 2023
@alexghr alexghr moved this from Todo to In Progress in A3 Nov 20, 2023
@alexghr alexghr changed the title Add persistent acrhiver store Add persistent archiver store Nov 20, 2023
This was referenced Nov 20, 2023
Closed
Merged
PhilWindle pushed a commit that referenced this issue Nov 28, 2023
@alexghr
feat: persistent archiver store (#3410)
4735bde 
This PR adds a new implementation for an archiver store that's backed by
LMDB. See #3361 for details.

Fix #3361
@github-project-automation github-project-automation bot moved this from In Progress to Done in A3 Nov 28, 2023
@AztecBot AztecBot mentioned this issue Nov 28, 2023
Merged
spypsy pushed a commit that referenced this issue Nov 28, 2023
@AztecBot
chore(master): Release 0.16.1 (#3435)
740024d 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-packages: 0.16.1</summary>

##
[0.16.1](aztec-packages-v0.16.0...aztec-packages-v0.16.1)
(2023-11-28)


### Features

* Added poseidon2 hash function to barretenberg/crypto
([#3118](#3118))
([d47782b](d47782b))
* Aztec CI files in Noir
([#3430](#3430))
([1621f3a](1621f3a))
* Persistent archiver store
([#3410](#3410))
([4735bde](4735bde)),
closes
[#3361](#3361)


### Bug Fixes

* **ci:** Don't leave DRY_DEPLOY unset
([#3449](#3449))
([454e316](454e316))
* **ci:** Publishing dockerhub manifests
([#3451](#3451))
([a59e7f0](a59e7f0))
* Hotfix noir sync
([#3436](#3436))
([c4e4745](c4e4745))


### Miscellaneous

* **docs:** Core concepts page in getting-started
([#3401](#3401))
([1a62f73](1a62f73))
* Point acir tests at noir master branch
([#3440](#3440))
([106e690](106e690))


### Documentation

* Further updates to the gas and fees whitepaper
([#3448](#3448))
([4152ba6](4152ba6))
* Updates to gas and fees yellow paper
([#3438](#3438))
([5f0e1ca](5f0e1ca))
</details>

<details><summary>barretenberg.js: 0.16.1</summary>

##
[0.16.1](barretenberg.js-v0.16.0...barretenberg.js-v0.16.1)
(2023-11-28)


### Miscellaneous

* **barretenberg.js:** Synchronize aztec-packages versions
</details>

<details><summary>barretenberg: 0.16.1</summary>

##
[0.16.1](barretenberg-v0.16.0...barretenberg-v0.16.1)
(2023-11-28)


### Features

* Added poseidon2 hash function to barretenberg/crypto
([#3118](#3118))
([d47782b](d47782b))


### Miscellaneous

* Point acir tests at noir master branch
([#3440](#3440))
([106e690](106e690))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Maddiaa0 pushed a commit that referenced this issue Nov 28, 2023
@alexghr @Maddiaa0
feat: persistent archiver store (#3410)
9b7f47c 
This PR adds a new implementation for an archiver store that's backed by
LMDB. See #3361 for details.

Fix #3361
AztecBot added a commit to AztecProtocol/barretenberg that referenced this issue Nov 29, 2023
@AztecBot
chore(master): Release 0.16.1 (#3435)
f49e8d8 
🤖 I have created a release *beep* *boop*
---


<details><summary>aztec-packages: 0.16.1</summary>

##
[0.16.1](AztecProtocol/aztec-packages@aztec-packages-v0.16.0...aztec-packages-v0.16.1)
(2023-11-28)


### Features

* Added poseidon2 hash function to barretenberg/crypto
([#3118](AztecProtocol/aztec-packages#3118))
([d47782b](AztecProtocol/aztec-packages@d47782b))
* Aztec CI files in Noir
([#3430](AztecProtocol/aztec-packages#3430))
([1621f3a](AztecProtocol/aztec-packages@1621f3a))
* Persistent archiver store
([#3410](AztecProtocol/aztec-packages#3410))
([4735bde](AztecProtocol/aztec-packages@4735bde)),
closes
[#3361](AztecProtocol/aztec-packages#3361)


### Bug Fixes

* **ci:** Don't leave DRY_DEPLOY unset
([#3449](AztecProtocol/aztec-packages#3449))
([454e316](AztecProtocol/aztec-packages@454e316))
* **ci:** Publishing dockerhub manifests
([#3451](AztecProtocol/aztec-packages#3451))
([a59e7f0](AztecProtocol/aztec-packages@a59e7f0))
* Hotfix noir sync
([#3436](AztecProtocol/aztec-packages#3436))
([c4e4745](AztecProtocol/aztec-packages@c4e4745))


### Miscellaneous

* **docs:** Core concepts page in getting-started
([#3401](AztecProtocol/aztec-packages#3401))
([1a62f73](AztecProtocol/aztec-packages@1a62f73))
* Point acir tests at noir master branch
([#3440](AztecProtocol/aztec-packages#3440))
([106e690](AztecProtocol/aztec-packages@106e690))


### Documentation

* Further updates to the gas and fees whitepaper
([#3448](AztecProtocol/aztec-packages#3448))
([4152ba6](AztecProtocol/aztec-packages@4152ba6))
* Updates to gas and fees yellow paper
([#3438](AztecProtocol/aztec-packages#3438))
([5f0e1ca](AztecProtocol/aztec-packages@5f0e1ca))
</details>

<details><summary>barretenberg.js: 0.16.1</summary>

##
[0.16.1](AztecProtocol/aztec-packages@barretenberg.js-v0.16.0...barretenberg.js-v0.16.1)
(2023-11-28)


### Miscellaneous

* **barretenberg.js:** Synchronize aztec-packages versions
</details>

<details><summary>barretenberg: 0.16.1</summary>

##
[0.16.1](AztecProtocol/aztec-packages@barretenberg-v0.16.0...barretenberg-v0.16.1)
(2023-11-28)


### Features

* Added poseidon2 hash function to barretenberg/crypto
([#3118](AztecProtocol/aztec-packages#3118))
([d47782b](AztecProtocol/aztec-packages@d47782b))


### Miscellaneous

* Point acir tests at noir master branch
([#3440](AztecProtocol/aztec-packages#3440))
([106e690](AztecProtocol/aztec-packages@106e690))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexghr alexghr

Labels
None yet
Projects
A3
Archived in project
Milestone
No milestone
1 participant
@alexghr