Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct Reporting of Multi-Threaded Test Outcomes #15

Open
wants to merge 12 commits into
base: 1_14_2_multithread
Choose a base branch
from

Conversation

mattjala
Copy link
Collaborator

@mattjala mattjala commented Nov 25, 2024

  • Rework test framework to aggregate and display multi-threaded results properly

This consists of the following major aspects:

  1. Consider a failure of a test/subtest in any thread to be an overall failure of that test/subtest
  2. Consider the number of tests to be constant regardless of the number of threads used to execute tests

Since subtests aren't exposed directly to testframe, several changes had to be made to support including them in the results while also avoid duplication during multithreading:

  • Number of subtests and test outcomes are now stored in the threadlocal info.

(Sub-)Test count and outcomes are recorded incrementally during runtime by TESTING/TESTING_2/H5_FAILED/SKIPPED/PASSED, and not displayed until after the subtest has finished in each thread. This is to prevent erroneous reporting of success in the case that it passes in the main thread and fails in another thread.

To pass information between testframe and the MT tests, some of these fields have been duplicated in the TestThreadArgs struct.

  • Subtest descriptions are stored on the threadlocal info and printed after the test is executed

All subtests are run in a row during test execution, so correct reporting requires waiting until after the subtests are complete to display the test information, which in turn requires storing the test description until it is to be displayed.

  • Change several TESTING calls in the I/O point selection test to TESTING_2 for compatibility with the new display code.

These probably should have been subtests in the first place, but it's now necessary to handle this right since the behavior between the top-level tests and subtest output is different.

  • Rework INCR_TEST_STAT into INCR_*_COUNT macros that manage the new threadlocal test/subtest information

  • Factor out the display portions of several test macros to use from testframe

  • Rename H5_API_TEST_FILENAME_MAX_LENGTH -> H5_TEST_FILENAME_MAX_LENGTH since it's no longer exclusive to the API tests

  • Suppress some special output in group/dataset tests on non-main threads

  • Replace non-atomic handling of potentially atomic num_errs_g

  • Known issue: If a subtest has any mid-test output, the output will be displayed before the subtest's description

This is a dependency for the following PRs:

MT VL Testing - mattjala#10
Make H5VL Multi-Thread Safe - mattjala#9
Implement virtual locks on IDs - mattjala#7

@mattjala mattjala marked this pull request as ready for review November 25, 2024 20:41
@mattjala mattjala force-pushed the mt_test_reporting branch 3 times, most recently from 9cf1378 to cc9369b Compare November 26, 2024 16:15
@mattjala mattjala marked this pull request as draft November 26, 2024 17:22
@mattjala mattjala marked this pull request as ready for review November 26, 2024 19:52
@mattjala mattjala force-pushed the mt_test_reporting branch 2 times, most recently from 6d97430 to 1a70c83 Compare December 30, 2024 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant