Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling flaky tests #490

Merged
merged 3 commits into from
Aug 4, 2023
Merged

Handling flaky tests #490

merged 3 commits into from
Aug 4, 2023

Conversation

glatterf42
Copy link
Member

@glatterf42 glatterf42 commented Aug 2, 2023

Closes #489 by marking tests as flaky. Through discussion with @khaeru, we agreed that this is the best option for now; should the tests make more issues in the future, we can still identify their root cause.

iiasa/message-ix-models#113 is waiting for this PR to be merged so that a temporary commit can be dropped there.

How to review

  • Read the diff and note that the CI checks all pass.

PR checklist

  • Continuous integration checks all ✅
  • Adapt CI tests; coverage checks both ✅

@glatterf42 glatterf42 added enh New features & functionality ci Continuous integration labels Aug 2, 2023
@glatterf42 glatterf42 self-assigned this Aug 2, 2023
@codecov
Copy link

codecov bot commented Aug 2, 2023

Codecov Report

Merging #490 (97f17da) into main (62a2a99) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #490   +/-   ##
=====================================
  Coverage   98.5%   98.5%           
=====================================
  Files         42      42           
  Lines       4489    4506   +17     
=====================================
+ Hits        4425    4442   +17     
  Misses        64      64           
Files Changed Coverage Δ
ixmp/backend/jdbc.py 95.2% <100.0%> (ø)
ixmp/tests/backend/test_jdbc.py 100.0% <100.0%> (ø)
ixmp/tests/test_access.py 100.0% <100.0%> (ø)
ixmp/tests/test_integration.py 100.0% <100.0%> (ø)
ixmp/tests/test_tutorials.py 100.0% <100.0%> (ø)

@glatterf42
Copy link
Member Author

The first attempt at windows-latest-py3.8 failed exactly with one of the errors that I hoped to avoid by marking tests as flaky and I can't find any sign in the logs that would indicate multiple reruns, which seems odd to me. Since GITHUB_ACTIONS should definitely be set in os.environ, I'm wondering whether sys.platform == "win32" is too restrictive. The recommended way to test for the platform might be different: import platform; platform.system() prints 'Windows', 'Linux', or 'Darwin'. I will try this tomorrow.

@glatterf42 glatterf42 requested a review from khaeru August 4, 2023 07:44
Copy link
Member

@khaeru khaeru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent! LGTM 🚀

@khaeru khaeru merged commit 5c55eec into iiasa:main Aug 4, 2023
@glatterf42 glatterf42 deleted the enh/ci/flaky-tests branch August 4, 2023 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous integration enh New features & functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Handling flaky tests
2 participants