[Concept] Add support for test case id tagging #66

PrzemekWirkus · 2016-01-15T14:09:27Z

Description

One binary can contain may test cases (test fixtures) with a unique test case ids.
We could add support to yield many test case id results per one test case execution.

Related PRs and changes:

Add support for test case id tagging - implementing #66 #67 (Added reporting, still JUnit reporting is not added)
[Concept] Add API for test case result handle htrun#58 (Added APIs and macros detection - test case result)
Testcase id results mbed-drivers#141 (Added new test_env APIs to handle new macros)

New success code for test case

New success code for test case is an integer with following properties:
Range of MBED_HOSTTEST_TESTCASE_FINISH() success:

+---------------+--------------+---------------+
|  success < 0  | success == 0 |  success > 0  |
+---------------+--------------+---------------+
| <------ ERROR |      OK      | FAIL -------> |
+---------------+--------------+---------------+

Example simple test case inside current test source code

// Somewhere in C++ test case file in your yotta module...
// My test case TESTCASE_NAME implementation
{
    int success = 0; // 0 for success! ;)
    MBED_HOSTTEST_TESTCASE_START(TESTCASE_NAME);

    // Do your testing bit here...

    // Make sure to set "success" to proper value:
    // 0 - 'success' result of the test case
    // >0 - 'failure' result of the test case
    // <0 - 'inconclusive' result of the test case

    MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,success);
}

You can think about this also like this:

if (result_of_my_test_case == 0) {
    // OK case !
    success = 0;
} else if (result_of_my_test_case > 0) {
    // FAILed case ! 
    // You can add any arbitrary and meaningful for you value < 0.
    // E.g. you can do success = 404; to mark in your test case 
    // that "server" returned 404 error.
    success = 404;    
} else if (result_of_my_test_case < 0) {
    // ERROR, test case execution was inconclusive
    // You can add any arbitrary and meaningful for you value > 0.
    // E.g. success = -1; when your test framework cough exception.
    success = -1;
}

Rationale

We need to move to new mode of work where one binary (test suite) can report multiple test cases results from automated run.

Changes

Added macros to mbed-drivers/test_env.h:

#define MBED_HOSTTEST_TESTCASE_START(TESTCASE)  notify_testcase_start(#TESTCASE)
#define MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE,SUCCESS)   notify_testcase_completion(#TESTCASE,SUCCESS)

Added macros implementation:

/** \brief Notifies test case start
  * \param Test Case ID name
  *
  * This function notifies test environment abort test case execution start.
  *
  */
void notify_testcase_start(const char *testcase_id)
{
    printf("{{testcase_start;%s}}" NL, testcase_id);
}

/** \brief Return partial (test case) result from test suite
  * \param Test Case ID name
  * \param Success code, 0 - success, >0 failure reason, <0 inconclusive
  *
  * This function passes partial test suite's test case result to test
  * environment.
  * Each test suite (in many cases one binary with tests) can return
  * multiple partial results used to track test case results.
  *
  * Test designers can use success code to return test case:
  * success == 0 - PASS, test case execution was successful.
  * success > 0  - FAILure, e.g. success == 404 can be used to
  *                pass "Server not found".
  * success < 0  - Inconclusive test case execution, e.g.
  *
  */
void notify_testcase_completion(const char *testcase_id, const int success)
{
    printf("{{testcase_finish;%s;%d}}" NL, testcase_id, success);
}

Added corresponding macros' equivalents on host test side:

selftest.testcase_start(testcase_name)
selftest.testcase_finish(testcase_name, testcase_result)

Example:

class RTCTest(BaseHostTest):
    def test(self, selftest):
        test_result = True
        selftest.testcase_start("RTC0001")
        # Test case execution
        selftest.testcase_finish("RTC0001", test_result)
        # ...

Example "STL" mbed-drivers test case source code with new macros in use

Note: We will modify in this example mbed-driver's STL.

We are adding two macros at the beginning of the test case and the end of it:

MBED_HOSTTEST_TESTCASE_START(STL001);
// test case execution
MBED_HOSTTEST_TESTCASE_FINISH(STL001, success_code);

Originally this test case was designed to run multiple checks and return one result, but now we can get multiple results from each test case and summary result from test suite passed via already used macro:

MBED_HOSTTEST_RESULT();

void runTest()
{
    int p_integers[] = {POSITIVE_INTEGERS};
    int n_integers[] = {NEGATIVE_INTEGERS};
    float floats[] = {FLOATS};
    bool result = true;

    MBED_HOSTTEST_TIMEOUT(10);
    MBED_HOSTTEST_SELECT(default_auto);
    MBED_HOSTTEST_DESCRIPTION(C++ STL);
    MBED_HOSTTEST_START("MBED_A3"); 

    // Test case STL001
    {
        MBED_HOSTTEST_TESTCASE_START(STL001);   // New macro
        std::vector<int> v_pints(p_integers, p_integers + TABLE_SIZE(p_integers));
        bool equal_result = std::equal(v_pints.begin(), v_pints.end(), p_integers);
        result = result && equal_result;
        MBED_HOSTTEST_TESTCASE_FINISH(STL001, !result); // New macro
    }

    // Test case STL002
    {
        MBED_HOSTTEST_TESTCASE_START(STL002);   // New macro
        const char* floats_str[] = {FLOATS_STR};
        float floats_transform[TABLE_SIZE(floats_str)] = {0.0};
        std::transform(floats_str, floats_str + TABLE_SIZE(floats_str), floats_transform, atof);
        bool equal_result = std::equal(floats_transform, floats_transform + TABLE_SIZE(floats_transform), floats);
        result = result && equal_result;
        MBED_HOSTTEST_TESTCASE_FINISH(STL002, !result); // New macro

        std::for_each(floats_str, floats_str + TABLE_SIZE(floats_str), printString());
        printf("\r\n");
        std::for_each(floats_transform, floats_transform + TABLE_SIZE(floats_transform), printFloat());
        printf("\r\n");
    }

Example host test for RTC test cases with new functions

In this examples we refactored test(self, selftest) function and added two new functions with two very similar test cases (we are only changing look count).

Changes:

We will still use return value from test(self, selftest) to give feedback from test suite.
We will add new two functions:

selftest.testcase_start("RTC0001")
# Your test case here...
selftest.testcase_finish("RTC0001", test_result)

Note that in this case test case is captured inside host test and not inside binary with mbed code.

class RTCTest(BaseHostTest):
    PATTERN_RTC_VALUE = "\[(\d+)\] \[(\d+-\d+-\d+ \d+:\d+:\d+ [AaPpMm]{2})\]"
    re_detect_rtc_value = re.compile(PATTERN_RTC_VALUE)

    def test(self, selftest):
        test_result = []
        test_result.append(self.test_rtc001(selftest))
        test_result.append(self.test_rtc002(selftest))
        return selftest.RESULT_SUCCESS if all(test_result) else selftest.RESULT_FAILURE

    def test_rtc001(self, selftest):
        test_result = True
        start = time()
        sec_prev = 0
        selftest.testcase_start("RTC0001")
        for i in range(0, 3):
            # Timeout changed from default: we need to wait longer for some boards to start-up
            c = selftest.mbed.serial_readline(timeout=10)
            if c is None:
                return selftest.RESULT_IO_SERIAL
            selftest.notify(c.strip())
            delta = time() - start
            m = self.re_detect_rtc_value.search(c)
            if m and len(m.groups()):
                sec = int(m.groups()[0])
                time_str = m.groups()[1]
                correct_time_str = strftime("%Y-%m-%d %H:%M:%S %p", gmtime(float(sec)))
                single_result = time_str == correct_time_str and sec > 0 and sec > sec_prev
                test_result = test_result and single_result
                result_msg = "OK" if single_result else "FAIL"
                selftest.notify("HOST: [%s] [%s] received time %+d sec after %.2f sec... %s"% (sec, time_str, sec - sec_prev, delta, result_msg))
                sec_prev = sec
            else:
                test_result = False
                break
            start = time()
        selftest.testcase_finish("RTC0001", test_result)
        return test_result

    def test_rtc002(self, selftest):
        test_result = True
        start = time()
        sec_prev = 0
        selftest.testcase_start("RTC0002")
        for i in range(0, 5):
            # Timeout changed from default: we need to wait longer for some boards to start-up
            c = selftest.mbed.serial_readline(timeout=10)
            if c is None:
                return selftest.RESULT_IO_SERIAL
            selftest.notify(c.strip())
            delta = time() - start
            m = self.re_detect_rtc_value.search(c)
            if m and len(m.groups()):
                sec = int(m.groups()[0])
                time_str = m.groups()[1]
                correct_time_str = strftime("%Y-%m-%d %H:%M:%S %p", gmtime(float(sec)))
                single_result = time_str == correct_time_str and sec > 0 and sec > sec_prev
                test_result = test_result and single_result
                result_msg = "OK" if single_result else "FAIL"
                selftest.notify("HOST: [%s] [%s] received time %+d sec after %.2f sec... %s"% (sec, time_str, sec - sec_prev, delta, result_msg))
                sec_prev = sec
            else:
                test_result = False
                break
            start = time()
        selftest.testcase_finish("RTC0002", test_result)
        return test_result

Example reporting

$ mbedgt -V -n mbed-drivers-test-rtc,mbed-drivers-test-stl,mbed-drivers-test-cstring
...
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226489a1e6c000000000000000000000000b543e3d4
mbedgt: test suite 'mbed-drivers-test-stl' ........................................................... OK in 2.91 sec
        test case 'STL001' ........................................................................... OK in 0.00 sec
        test case 'STL002' ........................................................................... OK in 0.01 sec
        test case 'STL003' ........................................................................... OK in 0.01 sec
        test case 'STL004' ........................................................................... OK in 0.00 sec
mbedgt: test suite report:
+---------------+---------------+---------------------------+--------+--------------------+-------------+
| target        | platform_name | test suite                | result | elapsed_time (sec) | copy_method |
+---------------+---------------+---------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | OK     | 8.71               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | FAIL   | 2.95               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | OK     | 2.91               | shell       |
+---------------+---------------+---------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 FAIL / 2 OK
mbedgt: test case report:
+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+
| target        | platform_name | test suite                | test case  | result | elapsed_time (sec) | copy_method |
+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | RTC0001    | OK     | 3.12               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc     | RTC0002    | OK     | 4.19               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS001 | OK     | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS002 | OK     | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS003 | OK     | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS004 | FAIL   | 0.0                | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS005 | FAIL   | 0.0                | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS006 | FAIL   | 0.0                | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS007 | FAIL   | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring | STRINGS008 | FAIL   | 0.0                | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL001     | OK     | 0.0                | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL002     | OK     | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL003     | OK     | 0.01               | shell       |
| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl     | STL004     | OK     | 0.0                | shell       |
+---------------+---------------+---------------------------+------------+--------+--------------------+-------------+
mbedgt: test case results: 5 FAIL / 9 OK
mbedgt: completed in 105.04 sec
mbedgt: exited with code 1

The text was updated successfully, but these errors were encountered:

ciarmcom · 2016-01-15T14:51:06Z

ARM Internal Ref: IOTSYST-768

PrzemekWirkus · 2016-01-15T16:31:56Z

DoD:

Documentation,
Macro on test case side (C/C++),
Support for greentea and macros for test case ids.

0xc0170 · 2016-01-18T07:35:01Z

I don't understand this feature, please can you provide C/C++ and python side code snippets for this test unit with many test cases ids? Is this to synchronize client and host, both directions?

Provided code snippet below does not set the full picture to me.
BaseHostTest.test method is blocking at the moment. Is this a step forward for async BaseHostTest.test ? Why do we need a name parameter? why selftest does not print the current running one? How do I assign the testaname ? Random number generator ?

class RTCTest(BaseHostTest):
    def test(self, selftest):
        test_result = True
        selftest.testcase_start("RTC0001")
        # Test case execution
        selftest.testcase_finish("RTC0001", test_result)
        # ...

Why testcase_start is not automatic (once test() is invoked), case start is indicated (it might be useful to have testcase_start manual method?), and testcase finish - does it need 2 arguments, why the selftest or self does not contain result , plus test id is automatic?

ID vs name? name is a test unit name, they match at the client side and host test side:

//client C/C++
MBED_HOSTTEST_SELECT(tcp_echo_client);

// python host
class RTCTest(BaseHostTest):
  name = 'name'

How to do that id? I found MBED_HOSTTEST_TESTCASE_START/FINISH macro, how to use those and on the host site to process those?

PrzemekWirkus · 2016-01-18T13:07:02Z

@0xc0170 Martin, I will provide and example, this description is not finished yet. I need to provide a guide and examples to fully cover this feature. I just had no time to finish this yesterday.

PrzemekWirkus · 2016-01-18T16:19:57Z

Reviewed this and other proposals today with Brendan, Martin and Niklas.

0xc0170 · 2016-01-19T06:59:58Z

MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,success);

we use result, not success? I saw in host tests results, and MBED_HOSTTEST_RESULT(RESULT)

From the client side (c/c++) as it is (2 new macros) is fine, but from the host side, how htrun knows how many test_ are there, is it like with test frameworks, that a method needs to start with test_ ?

def test_rtc001(self, selftest):
  # do magic

def test_rtc100(self, selftest):
  # do magic

selftest.testcase_start("RTC0002")

Wouldn't automatic naming make sense (test_ name), name would be upper case -> RTC001, RTC100 in my case. Or just a variable to set (same as for BaseHosttest, we are setting name, not calling a method), selftest.testcase.name = RTC001 (this is the name of the current test case, it starts once method test_rtc001 is invoked, finished once returned).
However, if the goal for this testcases is to be async in the future, then we might rethink this concept. What about TestCase class which would provide start(), finish(), set_timeout() ? I would like to know, what TestUnit (BaseHostTest I believe its named currently) provides (methods - test_, setup, teardown), and TestCase (start, finish, set_timeout).

PrzemekWirkus · 2016-01-19T09:49:09Z

MBED_HOSTTEST_RESULT(RESULT) macro is used to return result from test suite (binary with all tests).
MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,SUCCESS); macro is used to return 'success' from each test case inside test suite.

So your pseudo-code for one binary would look like:

int main() {

    {
        MBED_HOSTTEST_TESTCASE_START(TESTCASE_NAME);
        # test case code
        MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,SUCCESS);
    }
    {
        MBED_HOSTTEST_TESTCASE_START(TESTCASE_NAME);
        # test case code
        MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,SUCCESS);
    }
    ...
    {
        MBED_HOSTTEST_TESTCASE_START(TESTCASE_NAME);
        # test case code
        MBED_HOSTTEST_TESTCASE_FINISH(TESTCASE_NAME,SUCCESS);
    }

    MBED_HOSTTEST_RESULT(RESULT);
    return 0;
}

In above code you have three test cases execution and at the end you return test suite result.

PrzemekWirkus · 2016-01-19T09:51:11Z

I can imagine MBED_HOSTTEST_RESULT(RESULT) completely disheartening after we refactor whole macro and sync stuff. You probably remember our yesterday talk with Niklas and Brendan, we gonna issue ton of tickets to improve a lot this things.
But this is a step towards a good direction and backward compatibility with older tests.

PrzemekWirkus · 2016-01-19T09:51:59Z

Wouldn't automatic naming make sense (test_ name), name would be upper case -> RTC001, RTC100 in my case.
Yes it would, we discussed this yesterday with N. & B. You probably missed it :)

mazimkhan · 2016-01-19T10:29:12Z

Comment on grouping of tests. The example mbedgt test report gives idea:

That rtc and cstring binaries will have subtests. Is it true.
Does it also mean that /test//main.cpp will have sub tests?
Is the one binary multiple tests solution only for subtests?

My understanding of one binary having multiple tests was that one binary can have all or a group of tests based on the target memory size? and it will use clitest interface for test indexing, start and verdict.

niklarm · 2016-01-19T11:48:47Z

I wholeheartedly disagree with returning success as a signed integer!
Returning 404, or -42 as an integer does not make sense, when you can use printf to directly output a failure string.
What is the python side supposed to do with an integer? Will you first print a string that maps the error code to a failure string? This makes no sense.

What is the semantical difference between Error and Fail? Both are "not success" and therefore the test didn't pass.
Perhaps Error and Warning, where Error prints and fails, and Warning prints, but continues, might be a better scheme.
Then again: This is a (unit-) test, why would you have warning in there? And why would you ignore them?

Error codes are used because cheap to pass around (hence an int) and machine readable.
We don't need that. We need human readable, understandable error messages.
Why not just be verbose and print why it failed?

niklarm · 2016-01-19T12:02:34Z

selftest.testcase_start("RTC0001")
vs.
MBED_HOSTTEST_TESTCASE_START(STL002);

IT'S A ~~TRAP~~ STRING.
MBED_HOSTTEST_TESTCASE_START("STL002");

There is absolutely no reason to not use a string here. None.

mazimkhan · 2016-01-19T12:11:21Z

@niklas-arm regarding difference between Error and Fail. We need this to differentiate between between actual test failure and automation failure. Example the target wanted an input from host test side to carry on the test and failed on timeout. So not really a test failure.

I more inclined towards using word INCONCLUSIVE. This status would be like an error for for automation/system test team. Device software team only need to debug Failures.

However, I am not sure we should restrict detection of automation issues on host side only. So only PASS and FAIL from target?

mazimkhan · 2016-01-19T12:14:30Z

Also I advocate only three top level verdicts INCONC/ERROR (inconclusive/automation error), FAIL and PASS. While communicating INCONC/ERROR or FAIL the test can option pass relevent information in form of a string or error code that easily understandable in the report.

mazimkhan · 2016-01-19T12:20:06Z

test_rtc001 is a subtest proposal on host side. Do we need this? What if two sub tests in two different main tests need same logic. If a binary can print a list of tests coded in it and their corresponding host tests, mbed-host-test can load the required host test in the same fashion as it does now. Hence I don't see reason to implement this.

PrzemekWirkus · 2016-01-19T12:41:33Z

@mazimkhan We were till now able to conclude test (case) execution result on DUT and/or host test side.

DUT side - binary can conclude test case result itself and print {success}{end}
DUT or host test - For example in network test both sides can say "error" / "fail" - e.g. conenction can be dropped on one or the other side.
Host test - host test decides if test (case) passes / fails / errors - e.g. timer tests: DUT can't measure time on itself.

niklarm · 2016-01-19T13:15:38Z

@mazimkhan Ok, good point. I would be fine with an "automation" error, which signifies failures related to running the test, not the test itself.

PrzemekWirkus · 2016-03-04T16:42:25Z

Added in version 0.2.0.

ciarmcom added the mirrored label Jan 15, 2016

PrzemekWirkus added the enhancement label Jan 17, 2016

PrzemekWirkus mentioned this issue Jan 18, 2016

Add support for test case id tagging - implementing #66 #67

Closed

PrzemekWirkus changed the title ~~Add support for test case id tagging~~ [Feature] Add support for test case id tagging Jan 19, 2016

PrzemekWirkus changed the title ~~[Feature] Add support for test case id tagging~~ [Concept] Add support for test case id tagging Jan 20, 2016

mazimkhan mentioned this issue Jan 20, 2016

Serial output is cut off and test times out (on OS X) ARMmbed/htrun#25

Closed

PrzemekWirkus self-assigned this Feb 9, 2016

PrzemekWirkus closed this as completed Mar 4, 2016

ciarmcom added the closed_in_jira label Mar 4, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Concept] Add support for test case id tagging #66

[Concept] Add support for test case id tagging #66

PrzemekWirkus commented Jan 15, 2016

ciarmcom commented Jan 15, 2016

PrzemekWirkus commented Jan 15, 2016

0xc0170 commented Jan 18, 2016

PrzemekWirkus commented Jan 18, 2016

PrzemekWirkus commented Jan 18, 2016

0xc0170 commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

niklarm commented Jan 19, 2016

niklarm commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

niklarm commented Jan 19, 2016

PrzemekWirkus commented Mar 4, 2016

[Concept] Add support for test case id tagging #66

[Concept] Add support for test case id tagging #66

Comments

PrzemekWirkus commented Jan 15, 2016

Description

Related PRs and changes:

New success code for test case

Example simple test case inside current test source code

Rationale

Changes

Example "STL" mbed-drivers test case source code with new macros in use

Example host test for RTC test cases with new functions

Example reporting

ciarmcom commented Jan 15, 2016

PrzemekWirkus commented Jan 15, 2016

0xc0170 commented Jan 18, 2016

PrzemekWirkus commented Jan 18, 2016

PrzemekWirkus commented Jan 18, 2016

0xc0170 commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

niklarm commented Jan 19, 2016

niklarm commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

mazimkhan commented Jan 19, 2016

PrzemekWirkus commented Jan 19, 2016

niklarm commented Jan 19, 2016

PrzemekWirkus commented Mar 4, 2016