Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use verbose in ci to prioritize humans over machines #957

Closed
wants to merge 1 commit into from

Conversation

faddat
Copy link
Contributor

@faddat faddat commented May 13, 2023

Description

Makes the automated tests invoke:

go run ./tests/e2e/... --happy-path-only --verbose

This way we can look at the exact same results, and save human time.


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • Included the correct type prefix in the PR title
  • Added ! to the type prefix if API or client breaking change
  • Targeted the correct branch (see PR Targeting)
  • Provided a link to the relevant issue or specification
  • Followed the guidelines for building SDK modules
  • Included the necessary unit and integration tests
  • Added a changelog entry to CHANGELOG.md
  • Included comments for documenting Go code
  • Updated the relevant documentation or specification
  • Reviewed "Files changed" and left comments if necessary
  • Confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic
  • reviewed API design and naming
  • reviewed documentation is accurate
  • reviewed tests and test coverage

@faddat faddat requested a review from a team as a code owner May 13, 2023 23:21
@MSalopek
Copy link
Contributor

Hey, that's a valid point but I suggest refraining from doing so at this point.

  1. There's so much output that the tests in e2e CI will fail (some validators will miss blocks) -> there's an increase in false negatives.
  2. e2e tests as they stand now are timing sensitive, block times vary due to the fact that ther run in a very resource constrained environment
  3. reporting is basic and needs to be improved

The e2e framework is getting a major facelift, due to complete in Q2:

Main features:

  1. sensible, human readable reports for each CI run
  2. reducing resource usage
  3. optimizing happy path e2e tests for CI execution
  4. configurable execution paths
  5. regression tests between different ICS versions
  6. test case randomization & non-determinism tests in CI

The progress is tracked in this epic: cosmos/gaia#2406 and these issues are tracking current progress:

@mpoke
Copy link
Contributor

mpoke commented May 15, 2023

@faddat Thanks for your contribution. Closing this PR based on the comment above.

@mpoke mpoke closed this May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants