Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve assert output on failure #239

Merged
merged 1 commit into from
Nov 28, 2024
Merged

Improve assert output on failure #239

merged 1 commit into from
Nov 28, 2024

Conversation

jku
Copy link
Member

@jku jku commented Nov 22, 2024

  • Failures where the assert happened in Clientrunner.assert_metadata where not handled as pytest verbose asserts: Fix this by registering the module for assert rewriting
  • assert on serialized metadata so that pytest assert gives us nice output
  • Increase verbosity in Makefile so it's at same level as the action

Fixes #230, mostly

@jku
Copy link
Member Author

jku commented Nov 22, 2024

@loosebazooka this should help a little. I can't add you as reviewer but feel free.

>       assert trusted == b"", f"Unexpected trusted role {role} content"
E       AssertionError: Unexpected trusted role timestamp content
E       assert b'{\n "signat...on": 2\n }\n}' == b''
E         
E         Full diff:
E         - b''
E         + (b'{\n "signatures": [\n  {\n   "keyid": "e04d636e13f11a220ea4142d3950ac50b8ee'
E         +  b'6af91e95effbb3bee4df114d6b4e",\n   "sig": "304402206b30d8e293d3caefb7b81a'
E         +  b'6a5c3ed5cd7ed3a1e15d8df5e0da3cef56fdd6f38f02205c5b130c1403e3fba3babdba373711'
E         +  b'648dab5154c57e8e2fe634cb04698c126e"\n  },\n  {\n   "keyid": "ec9a5050bb6c78'...
E         
E         ...Full output truncated (9 lines hidden), use '-vv' to show

tuf_conformance/client_runner.py:138: AssertionError

Note that the pytest verbosity level affects how much diff pytest spits out:

  • at current verbosity level used by both Makefile and the action we get a handful of lines
  • Using "pytest -vv" gets you full diff I think -- but we don't currently expose verbosity level in either makefile or the action, you can only do that by manually invoking pytest

* Failures where the assert happened in Clientrunner.assert_metadata
  where not handled as pytest verbose asserts: Fix this by registering
  the module for assert rewriting
* assert on serialized metadata so that pytest assert gives us nice
  output
* Increase verbosity in Makefile so it's at same level as the action

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku jku force-pushed the assert-output-fixes branch from deedd3d to b2e8446 Compare November 28, 2024 15:06
@jku jku merged commit 457e9c1 into main Nov 28, 2024
6 checks passed
jku added a commit that referenced this pull request Nov 28, 2024
This release contains no new tests and test results are not expected to
change, except in one case: If `test_root_rotation` or
`test_non_root_rotations` are marked as "Expected To Fail" because of
the non-default keytype, this expectation can now be removed: these
tests now use the default keytype (ecdsa).

* local testing: "make dev" now checks for faketime binary
  to enable failing early (#240)
* Failure output was improved (#239)
* Tests now use default keytype whenever possible:
  In `practice test_non_root_rotations()` and `test_root_rotation()`
  switched from ed25519 to the default ecdsa key (#238)
* Expected failures are now documented better (#233)

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku jku mentioned this pull request Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Output diff or expected/actual?
1 participant