Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Extract parsing to its own pass and do it in parallel #4063

Merged
merged 7 commits into from
Jan 18, 2024

Conversation

sirasistant
Copy link
Contributor

@sirasistant sirasistant commented Jan 17, 2024

Description

Rework of #3849 after file_manager_with_stdlib was created. Also as @kevaundray suggested extracted the parsing above the context.

Resolves #3838

Problem*

Parsing is currently done when collecting, allowing only to parse files used, but making it very difficult to do parallel parsing or to have cached parsing.

Summary*

This PR extracts parsing to its own pass, and makes the Context take the parsed files. The creator of the context is the one in charge to do the parsing of the file manager, so Nargo, the LSP and wasm need to handle the parsed files. This PR uses rayon to do parallel parsing in Nargo & LSP. It reduces the time taken to process the on save notification in the LSP from ~700ms on protocol circuits to ~250ms.

With parsing being in its own pass, this opens the door for the LSP to cache parsing. It has access to the file manager and the parsed files, so it can detect which files have changed on disk and only parse those when necessary.

Additional Context

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [Exceptional Case] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

@sirasistant sirasistant marked this pull request as ready for review January 17, 2024 14:18
@sirasistant sirasistant mentioned this pull request Jan 18, 2024
5 tasks
Copy link
Contributor

@kobyhallx kobyhallx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 💯

@kevaundray kevaundray added this pull request to the merge queue Jan 18, 2024
Merged via the queue into master with commit 569cbbc Jan 18, 2024
30 checks passed
@kevaundray kevaundray deleted the arv/separate_layer_parsing branch January 18, 2024 13:32
github-merge-queue bot pushed a commit that referenced this pull request Jan 19, 2024
# Description
After #4063 we are able to cache
parsing results.

This PR showcases naive caching in the LSP
## Problem\*

Resolves <!-- Link to GitHub Issue -->

## Summary\*



## Additional Context



## Documentation\*

Check one:
- [ ] No documentation needed.
- [ ] Documentation included in this PR.
- [ ] **[Exceptional Case]** Documentation to be submitted in a separate
PR.

# PR Checklist\*

- [ ] I have tested the changes locally.
- [ ] I have formatted the changes with [Prettier](https://prettier.io/)
and/or `cargo fmt` on default settings.
github-merge-queue bot pushed a commit that referenced this pull request Jan 22, 2024
🤖 I have created a release *beep* *boop*
---


<details><summary>0.23.0</summary>

## [0.23.0](v0.22.0...v0.23.0)
(2024-01-22)


### ⚠ BREAKING CHANGES

* Ban nested slices
([#4018](#4018))
* Breaking changes from aztec-packages
([#3955](#3955))
* Rename Arithmetic opcode to AssertZero
([#3840](#3840))
* remove circuit methods from noir_wasm
([#3869](#3869))

### Features

* Add `assert_max_bit_size` method to `Field`
([#4016](#4016))
([bc9a44f](bc9a44f))
* Add `noir-compiler` checks to `aztec_macros`
([#4031](#4031))
([420a5c7](420a5c7))
* Add a `--force` flag to force a full recompile
([#4054](#4054))
([27a8e68](27a8e68))
* Add dependency resolver for `noir_wasm` and implement `FileManager`
for consistency with native interface
([#3891](#3891))
([c29c7d7](c29c7d7))
* Add foreign call support to `noir_codegen` functions
([#3933](#3933))
([e5e52a8](e5e52a8))
* Add MVP `nargo export` command
([#3870](#3870))
([fbb51ed](fbb51ed))
* Add support for codegenning multiple functions which use the same
structs in their interface
([#3868](#3868))
([1dcfcc5](1dcfcc5))
* Added efficient field comparisons for bn254
([#4042](#4042))
([1f9cad0](1f9cad0))
* Assert maximum bit size when creating a U128 from an integer
([#4024](#4024))
([8f9c7e4](8f9c7e4))
* Avoid unnecessary range checks by inspecting instructions for casts
([#4039](#4039))
([378c18e](378c18e))
* Breaking changes from aztec-packages
([#3955](#3955))
([5be049e](5be049e))
* Bubble up `Instruction::Constrain`s to be applied as early as
possible. ([#4065](#4065))
([66f5cdd](66f5cdd))
* Cached LSP parsing
([#4083](#4083))
([b4f724e](b4f724e))
* Comparison for signed integers
([#3873](#3873))
([bcbd49b](bcbd49b))
* Decompose `Instruction::Cast` to have an explicit truncation
instruction ([#3946](#3946))
([35f18ef](35f18ef))
* Decompose `Instruction::Constrain` into multiple more basic
constraints ([#3892](#3892))
([51cf9d3](51cf9d3))
* Docker testing flow
([#3895](#3895))
([179c90d](179c90d))
* Extract parsing to its own pass and do it in parallel
([#4063](#4063))
([569cbbc](569cbbc))
* Implement `Eq` trait on curve points
([#3944](#3944))
([abf751a](abf751a))
* Implement DAP protocol in Nargo
([#3627](#3627))
([13834d4](13834d4))
* Implement generic traits
([#4000](#4000))
([916fd15](916fd15))
* Implement Operator Overloading
([#3931](#3931))
([4b16090](4b16090))
* **lsp:** Cache definitions for goto requests
([#3930](#3930))
([4a2140f](4a2140f))
* **lsp:** Goto global
([#4043](#4043))
([15237b3](15237b3))
* **lsp:** Goto struct member inside Impl method
([#3918](#3918))
([99c2c5a](99c2c5a))
* **lsp:** Goto trait from trait impl
([#3956](#3956))
([eb566e2](eb566e2))
* **lsp:** Goto trait method declaration
([#3991](#3991))
([eb79166](eb79166))
* **lsp:** Goto type alias
([#4061](#4061))
([dc83385](dc83385))
* **lsp:** Goto type definition
([#4029](#4029))
([8bb4ddf](8bb4ddf))
* **lsp:** Re-add code lens feature with improved performance
([#3829](#3829))
([8f5cd6c](8f5cd6c))
* Optimize array ops for arrays of structs
([#4027](#4027))
([c9ec0d8](c9ec0d8))
* Optimize logic gate ACIR-gen
([#3897](#3897))
([926460a](926460a))
* Prefer `AcirContext`-native methods for performing logic operations
([#3898](#3898))
([0ec39b8](0ec39b8))
* Remove range constraints from witnesses which are constrained to be
constants ([#3928](#3928))
([afe9c7a](afe9c7a))
* Remove truncation from brillig casts
([#3997](#3997))
([857ff97](857ff97))
* Remove truncations which can be seen to be noops using type
information ([#3953](#3953))
([cc3c2c2](cc3c2c2))
* Remove unnecessary predicate from `Lt` instruction
([#3922](#3922))
([a63433f](a63433f))
* Simplify chains of casts to be all in terms of the original `ValueId`
([#3984](#3984))
([2384d3e](2384d3e))
* Simplify multiplications by `0` or `1` in ACIR gen
([#3924](#3924))
([e58844d](e58844d))
* Support for u128
([#3913](#3913))
([b4911dc](b4911dc))
* Support printing more types
([#4071](#4071))
([f5c4632](f5c4632))
* Sync `aztec-packages`
([#4011](#4011))
([fee2452](fee2452))
* Sync commits from `aztec-packages`
([#4068](#4068))
([7a8f3a3](7a8f3a3))
* Use singleton `WasmBlackBoxFunctionSolver` in `noir_js`
([#3966](#3966))
([10b28de](10b28de))


### Bug Fixes

* Acir gen doesn't panic on unsupported BB function
([#3866](#3866))
([34fd978](34fd978))
* Allow abi encoding arrays of structs from JS
([#3867](#3867))
([9b713f8](9b713f8))
* Allow abi encoding tuples from JS
([#3894](#3894))
([f7fa181](f7fa181))
* Allow ast when macro errors
([#4005](#4005))
([efccec3](efccec3))
* Allow lsp to run inside of a docker container
([#3876](#3876))
([2529977](2529977))
* Bit-shifts for signed integers
([#3890](#3890))
([6ddd98a](6ddd98a))
* Checks for cyclic dependencies
([#3699](#3699))
([642011a](642011a))
* **debugger:** Crash when stepping through locations spanning multiple
lines ([#3920](#3920))
([223e860](223e860))
* Don't fail if no tests and the user didn't provide a pattern
([#3864](#3864))
([decbd0f](decbd0f))
* Fix advisory issue in cargo-deny
([#4077](#4077))
([19baea0](19baea0))
* Fixing dark mode background on the CTA button
([#3882](#3882))
([57eae42](57eae42))
* Fixup exports from `noir_wasm`
([#4022](#4022))
([358cdd2](358cdd2))
* Handle multiple imports in the same file
([#3903](#3903))
([219423e](219423e))
* Hoist constraints on inputs to top of program
([#4076](#4076))
([447aa34](447aa34))
* Implement missing codegen for `BlackBoxFunc::EcdsaSecp256r1` in
brillig ([#3943](#3943))
([2c5eceb](2c5eceb))
* Improve `nargo test` output
([#3973](#3973))
([3ab5ff4](3ab5ff4))
* Make `constant_to_radix` emit a slice instead of an array
([#4049](#4049))
([5cdb1d0](5cdb1d0))
* Operator overloading & static trait method references resolving to
generic impls ([#3967](#3967))
([f1de8fa](f1de8fa))
* Preserve brillig entrypoint functions without arguments
([#3951](#3951))
([1111465](1111465))
* Prevent `Instruction::Constrain`s for non-primitive types
([#3916](#3916))
([467948f](467948f))
* Remove panic for adding an invalid crate name in wasm compiler
([#3977](#3977))
([7a1baa5](7a1baa5))
* Return error rather instead of panicking on invalid circuit
([#3976](#3976))
([67201bf](67201bf))
* Search all levels of struct nesting before codegenning primitive types
([#3970](#3970))
([13ae014](13ae014))
* Update generics docs to mention we have traits now
([#3980](#3980))
([c2acdf1](c2acdf1))


### Miscellaneous Chores

* Ban nested slices
([#4018](#4018))
([f8a1fb7](f8a1fb7))
* Remove circuit methods from noir_wasm
([#3869](#3869))
([12d884e](12d884e))
* Rename Arithmetic opcode to AssertZero
([#3840](#3840))
([836f171](836f171))
</details>

<details><summary>0.39.0</summary>

## [0.39.0](v0.38.0...v0.39.0)
(2024-01-22)


### ⚠ BREAKING CHANGES

* Breaking changes from aztec-packages
([#3955](#3955))
* Rename Arithmetic opcode to AssertZero
([#3840](#3840))
* Remove unused methods on ACIR opcodes
([#3841](#3841))
* Remove partial backend feature
([#3805](#3805))

### Features

* Aztec-packages
([#3754](#3754))
([c043265](c043265))
* Breaking changes from aztec-packages
([#3955](#3955))
([5be049e](5be049e))
* Remove range constraints from witnesses which are constrained to be
constants ([#3928](#3928))
([afe9c7a](afe9c7a))
* Speed up transformation of debug messages
([#3815](#3815))
([2a8af1e](2a8af1e))
* Sync `aztec-packages`
([#4011](#4011))
([fee2452](fee2452))
* Sync commits from `aztec-packages`
([#4068](#4068))
([7a8f3a3](7a8f3a3))


### Bug Fixes

* Deserialize odd length hex literals
([#3747](#3747))
([4000fb2](4000fb2))
* Return error rather instead of panicking on invalid circuit
([#3976](#3976))
([67201bf](67201bf))


### Miscellaneous Chores

* Remove partial backend feature
([#3805](#3805))
([0383100](0383100))
* Remove unused methods on ACIR opcodes
([#3841](#3841))
([9e5d0e8](9e5d0e8))
* Rename Arithmetic opcode to AssertZero
([#3840](#3840))
([836f171](836f171))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: TomAFrench <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate parsing all modules in a separate stage
3 participants