Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code coverage suite (rebased #848 with workspace restructure) #1008

Draft
wants to merge 54 commits into
base: devnet-ready
Choose a base branch
from

Conversation

ales-otf
Copy link
Contributor

@ales-otf ales-otf commented Nov 20, 2024

Description

Adds code coverage suite to the repo

  • find and parse all pallets dynamically and in parallel
  • code to import missing sections of pallets defined with #[import_section(..)] / #[pallet_section]
  • collect metadata on all extrinsics/calls in all pallets, in parallel
  • collect metadata on free functions defined within pallets (non-calls)
  • find all tests in the repo in parallel
  • find all benchmark tests (v2 syntax) in the repo in parallel
  • upgrade existing benchmarking v1 tests to v2 (only one file, quick)
  • add a quick little lint disallowing benchmarking v1 syntax and enable this lint in build.rs
  • collect metadata and coverage info for hooks, might be impractical, happy path would be if people commonly enter hooks manually within tests, then you can detect tests that do this and expect one covering each hook state, possibly for each pallet
  • analyze tests and generate coverage statistics based on what functions are/are not mentioned across all tests
  • display code coverage results in the terminal from the build script (using sam's build-print crate)
  • by default don't run code coverage unless RUN_COVERAGE env var is present (from within build.rs)

This PR also restructures the workspace by removing the root library and encapsulating logic from the root's build.rs into linting crate API. It adds build.rs scripts to run the lints independently for each crate.

The code-coverage is a binary, which you run using:

cargo run -p subtensor-code-coverage

The default crate in workspace is configured to be node-subtensor, so all commands go to this crate, if you don't pass -p <name> argument. It allows cargo run node, without specifying the package name.

Future work:

  • take function complexity into account (number of distinct terminal points = needs more coverage)
  • use conventional code coverage tools for everything that isn't a pallet and then roll this information into the final report
  • once 100% coverage is achieved, have CI fail if coverage drops below 100% and flag the specific functions/extrinsics/whatever that are untested
  • take function call graph into account -- if function or extrinsic A calls function or extrinsic B, then coverage on A at least partially covers B

Related Issue(s)

  • Closes #[issue number]

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Other (please describe):

Checklist

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have run cargo fmt and cargo clippy to ensure my code is formatted and linted correctly
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

sam0x17
sam0x17 previously approved these changes Nov 22, 2024
@sam0x17 sam0x17 marked this pull request as ready for review November 22, 2024 18:50
@sam0x17
Copy link
Contributor

sam0x17 commented Nov 25, 2024

CI failing btw

@ales-otf ales-otf changed the base branch from sam-test-coverage to devnet-ready November 25, 2024 21:08
@ales-otf ales-otf dismissed sam0x17’s stale review November 25, 2024 21:08

The base branch was changed.

@ales-otf ales-otf changed the title Restructure workspace Code coverage suite (rebased #848 with workspace restructure) Nov 25, 2024
@ales-otf
Copy link
Contributor Author

CI failing btw

yeah, it's not ready yet. I'm toggling it into draft.

@ales-otf ales-otf marked this pull request as draft November 25, 2024 21:18
@ales-otf ales-otf added blue team defensive programming, CI, etc no-spec-version-bump PR does not contain changes that requires bumping the spec version labels Nov 25, 2024
@ales-otf ales-otf force-pushed the chore/restructure-linting branch 3 times, most recently from 5a7caa5 to e165f94 Compare November 26, 2024 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blue team defensive programming, CI, etc no-spec-version-bump PR does not contain changes that requires bumping the spec version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants