-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improving WASM Smart Contract Debugging and Code Coverage in near-workspaces-rs #324
Comments
I do like the idea! Absolutely, having a code coverage would absolutely be desirable! I believe that the way workspaces crate works is that it is launching a local node (sandbox), deploys a compiled wasm and then sends transactions via JSON RPC. Thinking out loud here: a potential solution would be to measure "how much binary" was executed in each call and "what part" of it. Then, calculating a ratio of executed parts to all of the binary size would give at least an estimation on the code coverage. But it seems like it should be implemented in the sandbox itself, not in the workspaces crate. |
That is more or less how LLVM works - the thing is, WASM is an embedded environment, so normal rust tooling for LLVM doesn't work (it is a no profiler_builtins environment). This is where minicov comes in. But because the environment is even more restricted than that (runs in VM with size limits and no file write) it's extra tricky. |
Thanks for bringing it up! This makes total sense to me. minicov seems to be exactly what is needed, I believe it should be added as a feature to near-sdk-rs, so it can add instrumentation as part of the macro expansion, and dump the profiling results via env::log_str. The question, however, would be the size of that profiling data, as there is a limit of how much logs can handle, though we can consider raising the limits for the local sandbox instance of nearcore. @njelich Would you like to champion the implementation? I am happy to help you to find the relevant resources and integration points. |
I'd be happy to. Right now I think it may be over the limit the logs can handle, thats why I thought adding a file write for debugging would be useful. EDIT: I had a play around with setting it up - currently getting.
Which is purely due to the size of |
Actually, after a second thought, you can just write the data to the contract storage (i.e.
Exactly right 👍 |
Regarding running a custom near-sandbox - how would this be done in near-workspaces 0.7.0? Based on a read through of the code it seems impossible right now. |
Alright, I've got the tests running on a custom network using near-workspaces 0.8.0 (waiting for release). I managed to make enough progress to get to a new error.
Maybe the parallel test execution is causing problems... EDIT: Works - I've got my very first |
Okay, I've made some progress but I'm having issues with data limits - right now I have a view function that returns the coverage data, and it seems to be truncating its output (interesting behavior - silently truncates). @frol @shariffdev |
I'll take time to check the source to see if it's something that can be helped. |
@njelich I did not run into truncated output for large outputs, if it got too large it was |
Okay, I've got it figured out - there were some deserialization issues. Next step on my side would be figuring out how to plug in coverage measurement after every function call - for now I have a view function that I use to get coverage - that doesn't work in practice, because it only has its own environment for measurement, but really it should be injected every time there is a function call or view, within the same VM execution process. What would be the best way to do that? @shariffdev @frol |
Since logs/events are out, nothing comes to mind atm. Maybe a helper method? Thoughts @frol |
It would probably require changes to nearcore in that case, to plug into the VM? Or maybe I can write a derive for coverage that adds the function call the end of each existing function call with some output channel. |
@njelich I believe that plugging in your code through near-sdk-rs derives is the best way to go. Just for your reference, you can find what kind of code near-sdk-rs macro generates in the snapshots: https://github.com/near/near-sdk-rs/blob/ee5cf867741d6d0d4db15857609b9e9268cc9b32/near-sdk-macros/src/core_impl/code_generator/snapshots/no_args_no_return_mut.snap, so each struct method gets auxiliary |
So I would inject a bit of code before the return in the method_wrapper code. |
@njelich Yep, that is the right place to look into. |
I've noticed something interesting in the process of trying to modify the near-sdk. Editing the Cargo.toml near-sdk dependency to the revision corresponding to release 4.1.1 breaks builds.
Any idea what might be happening there? I get errors around types:
|
@njelich This error is usually an indicator of that two versions of the same crate were pulled in. Try using patch section in Cargo toml file instead of editing the dependency path. Also, keep in mind that near-sdk-rs master branch has some breaking changes and will be released as 5.0 soon |
@njelich Are you in touch with Hacken team? Hacken folks recently released: wasmcov. The project provides WASM-based code coverage. They have some docs for use on near. |
I'll update the README and link the docs. @njelich I found this link broken https://github.com/hknio/wasmcov |
@shariffdev It was private. |
1 similar comment
@shariffdev It was private. |
Hi @frol, is the plan still to include code coverage in near-workspaces or is wasmcov the way to go? |
Hacken team received a grant to integrate code coverage solution, but I have not heard from them lately |
I have left Hacken a while ago, but I believe @bbarwik is likely in charge of this task now. |
Hey @frol @bokobza and everyone else. I almost finished this tool for near, it's in beta phase. The current beta version is available only on github - https://github.com/hknio/wasmcov/ - I'll push it to creates.io by the end of the week. Here's how you can use it right now: To install use: To include it in your near project, add to dependencies: #[cfg(target_family = "wasm")]
wasmcov::near::add_coverage!(); Then in your integration tests, if you need to build WASM binaries first run To run your tests just use Then to generate coverage report run You need to have rust nightly version installed and llvm 18, you can get it from https://apt.llvm.org/ Please let me know what you think and if it worked correctly in your case. Here's an example report cross_contract.zip |
Issue Description
I am seeking to improve the debugging, profiling, and code coverage capabilities for WebAssembly (WASM) smart contracts using the
near-workspaces-rs
library. This issue serves as a discussion point for potential approaches to implement these features.One potential solution involves the use of
minicov
, a crate that provides code coverage and profile-guided optimization (PGO) forno_std
and embedded programs such as WASM smart contracts.Minicov
achieves this by utilizing a modified version of the LLVM profiling runtime, from which all dependencies on libc have been removed.However,
minicov
relies on outputting profraw files for its functionality. By default,near-workspaces-rs
and NEAR do not allow file writes as a feature, creating a barrier for usingminicov
in this context.Proposed Solution
To overcome this barrier, I propose that
near-workspaces-rs
or NEAR should allow a special debugging mode that permits file writes or some other output channel where these profraw files can be piped. This would enable the profraw files to be processed withllvm-cov
, thus allowing actual code coverage of WASM smart contracts on the NEAR runtime to be measured.This is particularly important as the general coverage developers can currently get is not actually on the NEAR runtime, and this caveat might be missed by those developing near smart contracts unfamiliar with Rust. For example, if only running unit tests, the
#[payable]
attributes might be completely missed on some functions that are thought to be tested based on default coverage reporting, but are missed in theworkspace-rs
tests.Discussion Points
I would like to open a broader discussion on this topic and outline other similar cases. Some potential discussion points could include:
near-workspaces-rs
or NEAR.I look forward to hearing the community's thoughts and ideas on this matter.
The text was updated successfully, but these errors were encountered: