tests/cmsis_rtos_v1 fails on nrf52840_pca10056

[spoorthik]

Describe the bug
test_kernel_systick fails on nrf52840_pca10056 in CI run and I couldn't reproduce locally. The issue is sporadic.

To Reproduce
Steps to reproduce the behavior:

1. mkdir build; cd build
2. cmake -DBOARD=nrf52840_pca10056
3. make flash
4. See error

Expected behavior
All test cases should pass without any assertion failure or crash.

Screenshots or console output

***** Booting Zephyr OS zephyr-v1.13.0-2205-g8d84f9229c (delayed boot 1000ms) *****
Running test suite test_cmsis_apis
===================================================================
starting test - test_kernel_start
PASS - test_kernel_start
===================================================================
starting test - test_kernel_systick

    Assertion failed at /home/jenkins/workspace/zephyr-master-ci-master/zephyr.git/tests/cmsis_rtos_v1/src/kernel_apis.c:51: test_kernel_systick: diff >= WAIT_TIME_US is false

FAIL - test_kernel_systick
===================================================================
starting test - test_thread_apis
PASS - test_thread_apis
===================================================================
starting test - test_thread_prio
PASS - test_thread_prio
===================================================================
starting test - test_thread_instances
PASS - test_thread_instances
===================================================================
starting test - test_timer
oneshot_callback (Timer 1) = 1
periodic_callback (Timer 2) = 1
periodic_callback (Timer 2) = 2
periodic_callback (Timer 2) = 3
periodic_callback (Timer 2) = 4
periodic_callback (Timer 2) = 5
periodic_callback (Timer 2) = 6
PASS - test_timer
===================================================================
starting test - test_mutex
PASS - test_mutex
===================================================================
starting test - test_mutex_lock_timeout
PASS - test_mutex_lock_timeout
===================================================================
starting test - test_semaphore
PASS - test_semaphore
===================================================================
starting test - test_mempool
PASS - test_mempool
===================================================================
starting test - test_mailq
PASS - test_mailq
===================================================================
starting test - test_messageq
PASS - test_messageq
===================================================================
starting test - test_signal_events_no_wait
PASS - test_signal_events_no_wait
===================================================================
starting test - test_signal_events_timeout
PASS - test_signal_events_timeout
===================================================================
starting test - test_signal_events_signalled
PASS - test_signal_events_signalled
===================================================================
starting test - test_signal_events_isr
PASS - test_signal_events_isr
===================================================================
Test suite test_cmsis_apis failed.
===================================================================
PROJECT EXECUTION FAILED

Environment (please complete the following information):

OS: Linux
Toolchain: Zephyr SDK 0.9.5
Since the failure is not seen locally, I couldn't git bisect to find on what commit the failure was first seen.

Additional context
#11721 #11722 also fails on nrf52840_pca10056 and they are sporadic.

[carlescufi]

@pizi-nordic can you take a look?

[pizi-nordic]

I was not able to reproduce this problem.
But I tried to play with the code a bit in order to trigger the error and I found a strange thing:

(...)
starting test - test_mutex_lock_timeout                 
***** MPU FAULT *****                                   
  Data Access Violation                                 
  MMFAR Address: 0xe                                    
***** Hardware exception *****                          
Current thread ID = 0x20000468                          
Faulting instruction address = 0xfbbc                   
Fatal fault in thread 0x20000468! Aborting.             
PASS - test_mutex_lock_timeout                          
========================================================
starting test - test_semaphore                          
PASS - test_semaphore                                   
========================================================
starting test - test_mempool                            
***** BUS FAULT *****                                   
  Imprecise data bus error                              
***** Hardware exception *****                          
Current thread ID = 0x20000518                          
Faulting instruction address = 0xf942                   
Fatal fault in thread 0x20000518! Aborting.             
PASS - test_mempool                                     
(...)
===================================================================
PROJECT EXECUTION SUCCESSFUL

Should we really consider this tests successful when we observe not expected faults?

[carlescufi]

Should we really consider this tests successful when we observe not expected faults?

No, not at all. We need to find out what triggered the fault and how to fix it.

[nashif]

Should we really consider this tests successful when we observe not expected faults?

In sanitycheck this should be caught and reported as failed.

[pizi-nordic]

@carlescufi: Fault was triggered by my changes in the code. What surprised me is that test with faults completed successfully.

@nashif: Is detection of fault is done in the external scripts (monitoring sanity check execution) or in C code running on device under test?

[galak]

@pizi-nordic can you open a new issue for the aspect of FAULTs not being treated as testcase fails.

Seems like we should close this issue as not-reproducible.

[pizi-nordic]

@galak: #12089

[nashif]

can't reproduce