-
Notifications
You must be signed in to change notification settings - Fork 6.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sanitycheck corrupted test case names in test-report.xml files #21162
Comments
Today I will run full sanitycheck cycle for that boards, to see if the error can be reproduced. |
We were parsing random FAIL messages from the output of test runs ad testcases and capturing them in the xml output. Now we only parse the name if it starts with test_. Fixes zephyrproject-rtos#21162 Signed-off-by: Anas Nashif <[email protected]>
moving up to medium, this bug is causing sanitycheck to produce inconsistent test results when something fails. |
I'm in process of checking your fix #21191 , I need time to verify it, to understand if it will help on 100%. |
We were parsing random FAIL messages from the output of test runs ad testcases and capturing them in the xml output. Now we only parse the name if it starts with test_. Fixes #21162 Signed-off-by: Anas Nashif <[email protected]>
If a test doesn't start with test_, twister currently emits an error: suite_name.test_name: Unknown status 'started' Following the suggestion by Maksim Masalski on PR#21191, change the regex to exclude spaces in the test name, and remove the check that the test_ prefix exists. See also zephyrproject-rtos#21162 Signed-off-by: Jeremy Bettis <[email protected]>
If a test doesn't start with test_, twister currently emits an error: suite_name.test_name: Unknown status 'started' Following the suggestion by Maksim Masalski on PR#21191, change the regex to exclude spaces in the test name, and remove the check that the test_ prefix exists. See also #21162 Signed-off-by: Jeremy Bettis <[email protected]>
If a test doesn't start with test_, twister currently emits an error: suite_name.test_name: Unknown status 'started' Following the suggestion by Maksim Masalski on PR#21191, change the regex to exclude spaces in the test name, and remove the check that the test_ prefix exists. See also zephyrproject-rtos#21162 Signed-off-by: Jeremy Bettis <[email protected]>
Describe the bug
Sometimes during the Sanitycheck I receive test results .xml files with the corrupted test case names. Inside of .xml file among test cases with proper names, can be detected names like that one below.
sys.mutex.nouser.thread_06@119. Failed to take mutex 0x20002288
To Reproduce
and running the sanitycheck for that test case and the board using
$sanitycheck -p qemu_cortex_m3 -T /home/maksim/zephyrproject/zephyr/tests/kernel/mutex/sys_mutex/
That bug is difficult to catch.
=========================XML ERROR1 PART START=============================
=========================XML ERROR1 PART END=============================
Additional comments
net.trickle.Trickle 1 0x80000c88 callback called
The most recent commit with that error was 42c5b0a
https://zephyrproject.testrail.io/index.php?/tests/view/14520629&group_by=cases:title&group_order=asc&group_id=-1
I can't reproduce for that commit that error too using
$sanitycheck -p mimxrt1050_evk -T /home/maksim/zephyrproject/zephyr/tests/net
When I ran sanitycheck, that test passed successfully.
You can try to find more corrupted test case names in the TestRail for the Zephyr RTOS
https://zephyrproject.testrail.io
Expected behavior
In normal conditions I expect even if test case fails to pass, error message in the Sanitycheck can't destroy test case name string in the test-report.xml file. Many other test cases also fail to pass, but their test case names not corrupted with the error message.
Environment:
The text was updated successfully, but these errors were encountered: