Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(lib/runtime/wasmer): common setupVM function #2685

Merged
merged 5 commits into from
Jul 26, 2022

Conversation

qdm12
Copy link
Contributor

@qdm12 qdm12 commented Jul 20, 2022

Changes

The main goal (not yet achieved) is to get the runtime version from the wasm code byte slice only.
This moves one step closer, by having a shared setupVM function taking in only the wasm code.

  • Fix: decompress WASM when checking runtime version
  • Fix: clear allocator when closing instance
  • Fix: add mutex protection for the entire UpdateRuntimeCode body
    • Add and use new close method on Instance (without mutex)
    • Use close method in Stop method
  • setupVM only takes in the WASM code bytes and returns an instance+allocator
  • WASM instance context data is set outside setupVM
  • Minor changes
    • Sentinel error for 'code is empty'
    • Remove useless debug log (just logs pointer addresses)
    • Instance named field mutex to find references easily
    • Explicit runtime Stop call in dot/state/initialize.go

Tests

Issues

#2418

Primary Reviewer

@timwu20

sync.Mutex
mutex sync.Mutex
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this was changed since I was having trouble finding references to the Lock/Unlock methods.

As a side note, I think we should avoid embedding things due to that, and also because it exports methods we might not want exported actually (like here, the mutex should be private).

@qdm12 qdm12 marked this pull request as ready for review July 20, 2022 19:38
@codecov
Copy link

codecov bot commented Jul 20, 2022

Codecov Report

Merging #2685 (2ee6693) into development (491af95) will decrease coverage by 0.16%.
The diff coverage is 45.31%.

@@               Coverage Diff               @@
##           development    #2685      +/-   ##
===============================================
- Coverage        62.82%   62.66%   -0.17%     
===============================================
  Files              211      211              
  Lines            27000    26992       -8     
===============================================
- Hits             16964    16915      -49     
- Misses            8489     8533      +44     
+ Partials          1547     1544       -3     
Flag Coverage Δ
unit-tests 62.66% <45.31%> (-0.17%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
lib/runtime/wasmer/instance.go 52.24% <44.44%> (-3.91%) ⬇️
dot/state/initialize.go 41.22% <100.00%> (+0.52%) ⬆️
dot/network/inbound.go 0.00% <0.00%> (-63.16%) ⬇️
dot/telemetry/mailer.go 56.71% <0.00%> (-4.48%) ⬇️
dot/network/light.go 85.25% <0.00%> (-0.80%) ⬇️
lib/grandpa/grandpa.go 60.81% <0.00%> (+0.33%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 491af95...2ee6693. Read the comment docs.

@qdm12 qdm12 force-pushed the qdm12/wasmer/setupvm-function branch from 649b9ad to 8375fdf Compare July 21, 2022 14:18
}

in.Lock()
defer in.Unlock()
in.ctx.Allocator = allocator // TODO we should no change the allocator of the parent instance
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line does the same thing as it did before.
It also looks like some very dangerous risky replacement, but it's changed in the next PR #2687 where CheckRuntimeVersion no longer depends on an existing instance.

- Fix: decompress WASM when checking runtime version
- `setupVM` only takes in the wasm code bytes and returns an instance+allocator
- WASM instance context data is set outside setupVM
- Sentinel error for 'code is empty'
- Remove useless debug log (just logs pointer addresses)
- OOS: name mutex field to find references easily
- Add `close` method on `Instance` (without mutex)
- Use `close` method in `Stop` method (thread safe)
@qdm12 qdm12 merged commit bdca089 into development Jul 26, 2022
@qdm12 qdm12 deleted the qdm12/wasmer/setupvm-function branch July 26, 2022 20:24
EclesioMeloJunior pushed a commit that referenced this pull request Aug 10, 2022
- Fix: decompress WASM when checking runtime version
- Fix: clear allocator when closing instance
- Fix: add mutex protection for the entire `UpdateRuntimeCode` body
  - Add and use new `close` method on `Instance` (without mutex)
  - Use `close` method in `Stop` method
- `setupVM` only takes in the WASM code bytes and returns an instance+allocator
- WASM instance context data is set outside setupVM
- Minor changes
  - Sentinel error for 'code is empty'
  - Remove useless debug log (just logs pointer addresses)
  - `Instance` named field `mutex` to find references easily
  - Explicit runtime `Stop` call in `dot/state/initialize.go`
rrtti pushed a commit to polkadot-fellows/seeding that referenced this pull request Sep 29, 2022
I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**.
I have been working full time on Gossamer since October 2021, mostly on the state trie and storage.
I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository.

I am requesting to join the Fellowship at rank 1.

## Main contributions

### Gossamer

- Fix memory leaks
  - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009)
  - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286)
  - Fix sync benchmark [#2234](ChainSafe/gossamer#2234)
- Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661))
- Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370))
- State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272))
- State trie fixes and improvements
  - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223)
  - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384)
  - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725)
  - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081)
- Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485))
- Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991)

Ongoing work:

- State trie lazy loading and caching
- State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530))

### Polkadot specification

➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)
@github-actions
Copy link

🎉 This PR is included in version 0.7.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants