Added minitest-retry gem to retry flapping tests that are able to pass through retries #3044
Conversation
…t from preservation queue rake task when running tests
ConnorSheremeta
requested review from
jefferya,
lagoan and
pgwillia
as code owners
| @@ -125,6 +125,7 @@ group :test do | |||
|
|
|||
| gem 'json-schema', '~> 3.0.0' | |||
| gem 'launchy' | |||
| gem 'minitest-retry', require: false | |||
There was a problem hiding this comment.
Nice find! Does it add much more time to running our test suites?
There was a problem hiding this comment.
It's hard to say as the time it takes for the tests to run varies quite a lot run to run on CI. It looks like it is taking around as long as it used to without this gem but it would require further testing to know for sure.
| collection_and_community_count = Community.count + Collection.count | ||
| assert_equal collection_and_community_count, | ||
| RedisClient.current.zcard(Rails.application.secrets.preservation_queue_name) | ||
| $stdout.stub(:puts, nil) do |
There was a problem hiding this comment.
That's a tidy workaround for the extra output. What do you think about adding a comment explaining why this block is here?
There was a problem hiding this comment.
Did some refactoring and added a comment in my most recent commits
| AdminUsersIndexTest#test_should_be_able_to_autocomplete_by_name | ||
| BatchIngestTest#test_invalid_without_files | ||
| DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully | ||
| DraftControllerTest#test_should_not_be_able_to_update_a_draft_item_when_saving_upload_files_form_that_has_no_file_attachments | ||
| DraftControllerTest#test_should_not_be_able_to_update_a_draft_thesis_when_saving_upload_files_form_that_has_no_file_attachments | ||
| ItemListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item | ||
| ThesisListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item |
There was a problem hiding this comment.
Are we tracking these in a ticket to review at some point? This list is longer than I was expecting. I recognize DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully
There was a problem hiding this comment.
I've added a ticket for this here
There was a problem hiding this comment.
I added the failure message in case we don't get back to it before the log becomes inaccessible after 90 days.
There was a problem hiding this comment.
Good catch, thank you!
I also removed the output from the preservation queue rake task that was polluting the test suite output.
Here are instances of these tests flapping in CI for further reference:
BatchIngestTest#test_invalid_without_files
DraftControllerTest#test_should_not_be_able_to_update_a_draft_item_when_saving_upload_files_form_that_has_no_file_attachments
DraftControllerTest#test_should_not_be_able_to_update_a_draft_thesis_when_saving_upload_files_form_that_has_no_file_attachments
DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully
AdminUsersIndexTest#test_should_be_able_to_autocomplete_by_name
ThesisListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item
ItemListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item
Will be creating an issue for resolving these flapping tests and link it here