Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitycheck corrupted test case names in test-report.xml files #21162

Closed
maksimmasalski opened this issue Dec 4, 2019 · 3 comments · Fixed by #21191
Closed

Sanitycheck corrupted test case names in test-report.xml files #21162

maksimmasalski opened this issue Dec 4, 2019 · 3 comments · Fixed by #21191
Assignees
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug

Comments

@maksimmasalski
Copy link
Collaborator

maksimmasalski commented Dec 4, 2019

Describe the bug
Sometimes during the Sanitycheck I receive test results .xml files with the corrupted test case names. Inside of .xml file among test cases with proper names, can be detected names like that one below.
sys.mutex.nouser.thread_06@119. Failed to take mutex 0x20002288

To Reproduce

  1. It was a sanitycheck run for the board qemu_cortex_m3 during the master build master-2019-11-10-53f30bc3ac
  2. I was not able to reproduce that bug again using the same commit 53f30bc
    and running the sanitycheck for that test case and the board using $sanitycheck -p qemu_cortex_m3 -T /home/maksim/zephyrproject/zephyr/tests/kernel/mutex/sys_mutex/
    That bug is difficult to catch.
  3. Original output test-report.xml file with that error

=========================XML ERROR1 PART START=============================

<testcase classname="qemu_cortex_m3:sys.mutex.nouser" name="sys.mutex.nouser.thread_06@119. Failed to take mutex 0x20002288" time="8.243418"><failure message="failed" type="failure">***** Booting Zephyr OS build zephyr-v2.0.0-1958-g53f30bc3ac24 *****
Running test suite mutex_complex
===================================================================
starting test - test_mutex
starting test - Test kernel Mutex API
===================================================================
Done LOCKING!  Current priority = 5

    Assertion failed at ZEPHYR_BASE/tests/kernel/mutex/sys_mutex/src/main.c:306: test_mutex: (rv not equal to 6)
thread_05 timed out and out priority should drop.

FAIL - test_mutex
===================================================================
starting test - test_user_access
SKIP - test_user_access
===================================================================
starting test - test_supervisor_access
PASS - test_supervisor_access
===================================================================
Test suite mutex_complex failed.
===================================================================
PROJECT EXECUTION FAILED
FAIL - thread_06@119. Failed to take mutex 0x20002288
</failure></testcase>

=========================XML ERROR1 PART END=============================

Additional comments

  1. Recently on November 29th the same error happened with the net.trickle for board mimxrt1050_evk.
    net.trickle.Trickle 1 0x80000c88 callback called
    The most recent commit with that error was 42c5b0a
    https://zephyrproject.testrail.io/index.php?/tests/view/14520629&group_by=cases:title&group_order=asc&group_id=-1
    I can't reproduce for that commit that error too using $sanitycheck -p mimxrt1050_evk -T /home/maksim/zephyrproject/zephyr/tests/net
    When I ran sanitycheck, that test passed successfully.

You can try to find more corrupted test case names in the TestRail for the Zephyr RTOS
https://zephyrproject.testrail.io

Expected behavior
In normal conditions I expect even if test case fails to pass, error message in the Sanitycheck can't destroy test case name string in the test-report.xml file. Many other test cases also fail to pass, but their test case names not corrupted with the error message.

Environment:

@maksimmasalski maksimmasalski added the bug The issue is a bug, or the PR is fixing a bug label Dec 4, 2019
@maksimmasalski
Copy link
Collaborator Author

maksimmasalski commented Dec 4, 2019

Today I will run full sanitycheck cycle for that boards, to see if the error can be reproduced.

@aescolar aescolar added area: Sanitycheck Sanitycheck has been renamed to Twister priority: low Low impact/importance bug labels Dec 4, 2019
nashif added a commit to nashif/zephyr that referenced this issue Dec 4, 2019
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.

Fixes zephyrproject-rtos#21162

Signed-off-by: Anas Nashif <[email protected]>
@nashif nashif added priority: medium Medium impact/importance bug and removed priority: low Low impact/importance bug labels Dec 5, 2019
@nashif
Copy link
Member

nashif commented Dec 5, 2019

moving up to medium, this bug is causing sanitycheck to produce inconsistent test results when something fails.

@nashif nashif added the has-pr label Dec 5, 2019
@maksimmasalski
Copy link
Collaborator Author

moving up to medium, this bug is causing sanitycheck to produce inconsistent test results when something fails.

I'm in process of checking your fix #21191 , I need time to verify it, to understand if it will help on 100%.

nashif added a commit that referenced this issue Dec 9, 2019
We were parsing random FAIL messages from the output of test runs ad
testcases and capturing them in the xml output. Now we only parse the
name if it starts with test_.

Fixes #21162

Signed-off-by: Anas Nashif <[email protected]>
jeremybettis added a commit to jeremybettis/zephyr that referenced this issue Aug 28, 2023
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'

Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.

See also zephyrproject-rtos#21162

Signed-off-by: Jeremy Bettis <[email protected]>
carlescufi pushed a commit that referenced this issue Aug 29, 2023
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'

Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.

See also #21162

Signed-off-by: Jeremy Bettis <[email protected]>
piotrnarajowski pushed a commit to piotrnarajowski/zephyr that referenced this issue Aug 31, 2023
If a test doesn't start with test_, twister currently emits an error:
suite_name.test_name: Unknown status 'started'

Following the suggestion by Maksim Masalski on PR#21191, change the
regex to exclude spaces in the test name, and remove the check that the
test_ prefix exists.

See also zephyrproject-rtos#21162

Signed-off-by: Jeremy Bettis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants