Document the tests we need for each consensus rule

[dconnolly]

Tasks

1. Create a draft list of tests that we need for each consensus rule
2. Create a draft test checklist for the default PR template
3. Review these drafts with the team
4. Merge the test list as a page in the Zebra dev book (or as an RFC?)
5. Link to the new test list from the Zebra dev book and the RFC template

[dconnolly]

Related to #395

[dconnolly]

See also, GitHub issue and PR templates

[mpguerra]

Should we defer this one until after First Alpha Release?

[mpguerra]

@ZcashFoundation/zebra-team Does it make sense to do this collaboratively during a meeting? At least "Agree on a set of tests we need for each consensus rule"

[teor2345]

Does it make sense to do this collaboratively during a meeting? At least "Agree on a set of tests we need for each consensus rule"

I think we need a specific proposal to have a productive discussion. Otherwise the discussion ends up too vague, or repeats decisions we've already made.

Based on our previous discussions, we've made updates to the PR template and RFC template, and created the Zebra Validation Matrix.

Someone needs to do the work to come up with a short, consistent list of tests. And summarise that list for the PR template. Then we can discuss their draft changes.

Here's what we have so far:

PR Template

The code in this pull request has:

• Documentation Comments
• Unit Tests and Property Tests

https://github.com/ZcashFoundation/zebra/blob/main/.github/pull_request_template.md

RFC Template - Test Plan

Explain how the feature will be tested, including:

• tests for consensus-critical functionality
• existing test vectors, if available
• Zcash blockchain block test vectors (specify the network upgrade, feature, or block height and network)
• property testing or fuzzing

The tests should cover:

• positive cases: make sure the feature accepts valid inputs
  • using block test vectors for each network upgrade provides some coverage of valid inputs
• negative cases: make sure the feature rejects invalid inputs
  • make sure there is a test case for each error condition in the code
  • if there are lots of potential errors, prioritise:
    • consensus-critical errors
    • security-critical errors, and
    • likely errors
• edge cases: make sure that boundary conditions are correctly handled

https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/rfcs/0000-template.md#test-plan

Zebra Validation Matrix

ZIP/Spec Unit Tests

• Accepts Valid Input
• Rejects Invalid Input
• Handles Edge Cases and Transitions Correctly

zcashd Implementation Compatibility

• Accepts Mainnet/Testnet Blocks
• Matches zcashd Rejections

https://docs.google.com/spreadsheets/d/10F5U9pkbQqwRkncqzFy5PnM6zRzJMjZF72X8XNSqVZw/edit#gid=0

[teor2345]

So for example, we might end up with:

List of Consensus Rule Tests

ZIP/Spec Tests

• Accepts Valid Input
• Rejects Invalid Input
• Handles Edge Cases and Transitions Correctly
• Matches Test Vectors (if any)
• Passes Property Tests (if feasible)

zcashd Implementation Compatibility

• Accepts Mainnet/Testnet Blocks
• Matches zcashd Rejections

Pull Request Template

• Documentation Comments
• Consensus Rule Tests
  See (link to full list of consensus rule tests)
• Tests for New or Changed Features

[dconnolly]

I think this issue is defunct