[devel_transport] Async host test execition support

[PrzemekWirkus]

Description

Async host test execution feature:

• Add KiVi transport protocol support - draft
• Removed compatibility with option --digest <stdin> and --digest <filename> (This may break yotta test command).
• Add test case reporting support.

Related PRs:

• Async greentea-client support
• Async features
• Async utest port
• greentea-client - extracted from mbed-drivers test_env.* APIs.

Design

• Simple key-value protocol is introduced. It is used to communicate between DUT and host. Protocol main features:
  • Protocol introduced is master-slave protocol, where master is host and slave is device under test.
  • Transport layer consist of simple {{KEY;VALUE}}} messages sent by slave (DUT). Both key and value are strings.
  • DUT always (except for handshake phase) initializes communication by sending key-value message to host.
  • To avoid miscommunication between master and slave simple handshake protocol is introduces:
    • Master (host) sends sync packet: {{__sync;UUID-STRING}}} with message value containing random UUID string.
    • DUT waits for {{__sync;...}} message in input stream and replies with the same packer {{__sync;...}}.
    • After correct sync packet is received by master, messages {{__timeout;%d}} and {{__host_test_name}} are expected.
  • Host parses DUTs tx stream and generates events sent to host test.
  • Each event is a tuple of (key, value, timestamp), where key and value are extracted from message and
• Host tests are now driven by simple async feature. Event state machine on master side is used to process messages from DUT. Each host test is capable of registering callbacks, functions which will be executed when event occur.
• DUT side uses simple parser to parse key-value pairs from stream. All non key-value dta will be ignored. parsing is blocking.
  • Message parsing transforms key-value string to Python event in this order:
    • {{key;value}} string captured on DUT output.
  • key-value data becomes a recognizable message with key (string) and value (string) payload.
  • Event is formed in host test, a tuple of key (string), value (string), timestamp where timestamp``` is time of message reception in Python time.time() format (float, time in seconds since the epoch as a floating point number.).
• Each host test registers callbacks for available events
• Few keys in key-value messaging protocol are promoted to be considered 'system events'.
  • User can't register to handle reserved events with few exceptions.
  • Reserved event/message keys have leading __ in name:
    • __sync - sync message, used by master and DUT to handshake.
    • __timeout - timeout in sec, sent by DUT after {{sync;UUID}} is received.
    • __host_test_name - host test name, sent by DUT after {{sync;UUID}} is received.
    • __notify_prn - sent by host test to print log message.
    • __notify_conn_lost - sent by host test's connection process to notify serial port connection lost.
    • __notify_complete - sent by DUT, async notificaion about test case result (true, false, none).
    • __coverage_start - sent by DUT, coverage data.
    • __testcase_start - sent by DUT, test case start data.
    • __testcase_finish - sent by DUT, test case result.
    • __exit - sent by DUT, test suite execution finished.
  • Non-Reserved event/message keys have leading __ in name:
    • __rxd_line - Event triggered when \n was found on DUT RXD channel. It can be overridden (self.register_callback('__rxd_line', <callback_function>)) and used by user. Event is sent by host test to notify a new line of text was received on RXD channel. __rxd_line event payload (value) in a line of text received from DUT over RXD.
• Each host test (master side) has four functions used by async framework:
  • setup() used to initialize host test and register callbacks.
  • result() used to return test case result when notify_complete() is not called.
  • teardown() used to finalize and resource freeing. It is guaranteed that teardown() will be always called after timeout or async test completion().
  • notify_complete(result : bool) used by host test to notify test case result. This result will be read after test suite TIMEOUTs or after DUT send __exit message (test suite execution finished event).
  • self.send_kv(key : string, value : string) - send key-value message to DUT.
  • self.log(text : string) - send event __notify_prn with text as payload (value). Your message will be printed in log.
• Result returned from host test is a test suite result. Test cases results are reported by DUT, usually using modified utest framework.

Callback

You can register callbacks in setup() phase or decorate callback functions using @event_callback decorator.

Callback registration in setup() method

from mbed_host_tests import BaseHostTest

class DetectRuntimeError(BaseHostTest):

    __result = False

    def callback_some_event(self, key, value, timeout):
        # Do something with 'some_event'
        pass

    def setup(self):
        # Reagister call back for 'some_event' event
        self.register_callback('some_event', self.callback_some_event)

    def result(self):
        # Do some return calculations
        return self.__result

Below the same callback registered using decorator:

Callback decorator definition

from mbed_host_tests import BaseHostTest

class DetectRuntimeError(BaseHostTest):

    __result = False

    @event_callback('some_event')
    def callback_some_event(self, key, value, timeout):
        # Do something with 'some_event'
        pass

    def setup(self):
        # Do some extra setup if required
        # You can also register here callbacks using self.register_callback(...) method
        pass

    def result(self):
        # Do some return calculations
        return self.__result

Parsing text received from DUT (line by line)

Example of host test expecting Runtime error ... CallbackNode ... string in DUT output.
We will use allowed to override __rxd_line event to hook to DUT RXD channel lines of text.

Before Greentea v0.2.0

from sys import stdout
from mbed_host_tests import BaseHostTest

class DetectRuntimeError(BaseHostTest):

    name = 'detect_runtime_error'

    def test(self, selftest):
        result = selftest.RESULT_FAILURE
        try:
            while True:
                line = selftest.mbed.serial_readline()

                if line is None:
                    return selftest.RESULT_IO_SERIAL

                stdout.write(line)
                stdout.flush()

                line = line.strip()

                if line.startswith("Runtime error") and line.find("CallbackNode") != -1:
                    result = selftest.RESULT_SUCCESS
                    break

        except KeyboardInterrupt, _:
            selftest.notify("\r\n[CTRL+C] exit")
            result = selftest.RESULT_ERROR

        return result

Using __rdx_line event

from mbed_host_tests import BaseHostTest

class DetectRuntimeError(BaseHostTest):
    """! We __expect__ to detect 'Runtime error' """

    __result = False

    def callback__rxd_line(self, key, value, timeout):
        # Parse line of text received over e.g. serial from DUT
        line = value.strip()
        if line.startswith("Runtime error") and "CallbackNode" in line:
            # We've found exepcted "Runtime error" string in DUTs output stream
            self.notify_complete(True)

    def setup(self):
        # Force, we force callback registration even it is a restricted one (starts with '__')
        self.register_callback('__rxd_line', self.callback__rxd_line, force=True)

    def result(self):
        # We will return here (False) when we reach timeout of the test
        return self.__result

    def teardown(self):
        pass

htrun new log format:

• [timestamp][source][level] - new log format, where:
  • timestamp - returned by Python's time.time().
  • source - log source.
    • CONN - connection process (pooling for connection source e.g. serial port),
    • SERI - serial port wrapper with standard read, write, flush interface,
    • HTST - host test object, HostTestBase derived object,
  • level - logging level:
    • INF (info),
    • WRN (warning) and
    • ERR (error).
    • TXD (host's TX channel, to DUT).
    • RXD (host's RX channel, from DUT).

Log example

• ``[1455218713.87][CONN][RXD] {{__sync;a7ace3a2-4025-4950-b9fc-a3671103387a}}`:
• Logged from CONN (connection process).
• RXD channel emitted {{__sync;a7ace3a2-4025-4950-b9fc-a3671103387a}}.
• Time stamp: 2016-02-11 19:53:27, see below:

>>> from time import strftime, gmtime
>>> t = 1455220407.20
>>> strftime("%Y-%m-%d %H:%M:%S", gmtime(t))

Greentea client API (currently test_env.h/cpp)

• Key-value formatted

void greentea_send_kv(const char *key, const char *value);
void greentea_send_kv(const char *key, const int value);

Functions are used to send key-string or key-integer value messages to master. This functions should replace typical printf() calls with payload/control data to host.

• Blocking wait for key-value pair message in input stream:

int greentea_parse_kv(char *out_key, char *out_value, const int out_key_len, const int out_value_len);

This function should replace scanf() used to check for incoming messages from master.
Function parses input and if key-value message is found load to out_key, out_value key-value pair. Use out_key_size and `out_value_size`` to define out buffers max size (including trailing zero).

Key-value transport protocol sequence

DUT                     host
---                     ----
 |    {{__sync;UUID}}     |
 |<-----------------------|
 |                        |
 |    {{__sync;UUID}}     |
 |----------------------->|

 - - - - - - - - - - -  - -
        HANDSHAKE !
 - - - - - - - - - - -  - -

 |                        |
 |   {{__timeout;%s}}     |
 |----------------------->|
 |                        |
 |{{__host_test_name;%s}} |
 |----------------------->|

 - - - - - - - - - - -  - -
 TEST SUITE CONTROL UNDER 
         HOST TEST 
 - - - - - - - - - - -  - -     
                                  ht = HostTest(__host_test_name)
 |                        |      ----     
 | {{key1;value}}         |       |
 | -------------------->  |       |  ht.setup()
 |          .             |       |  
 |          .             |       - - - - - - - - - -
 |          .             |        Host test callbacks registered    
 |          .             |       - - - - - - - - - - 
 | {{key2;value}}         |       |  
 | -------------------->  |       |  
 |                        |       |  
 |                        |       |  
 |                        |       | ht.callbacks[key1](key, value, timestamp)
 |                        |       |<----------------
 |                        |       | ht.callbacks[key2](key, value, timestamp)
 |                        |       |<----------------
 |                        |       |
 |                        |       |
 - - - - - - - - - - -  - - - - - -
          TEST CASE FLOW
 - - - - - - - - - - -  - - - - - -
 |                        |       |
 |                        |       | ht.notify_complete(true)
 |                        |       | (sets test suite 'result' to true
 |                        |       |<----------------
 |                        |       |
 - - - - - - - - - - -  - - - - - -
 - - - - - - - - - - -  - - - - - -
 |                        |       |
 | {{end;success}}        |       |  
 | -------------------->  |       | 
 |                        |       |
 | {{__exit;%d}}          |       |  
 | -------------------->  |       | 
 - - - - - - - - - - -  - - - - - -
  DUT'S TEST SUITE ENDED
  CONSUME REMAINING EVENTS
 - - - - - - - - - - -  - - - - - -
 |                        |       |
 |                        |       | result = ht.result()
 |                        |       |<----------------
 |                        |       |
 |                        |       | ht.teardown()
 |                        |       |<----------------
 |                        |       |
 |                        |       |

DUT - host test case workflow

DUT implementation

int main() {
    // 1. Handshake between DUT and host and
    // 2. Send test case related data
    GREENTEA_SETUP(15, "gimme_auto");  // __timeout, __host_test_name

    // ...
    // Send to master {{gimme_something; some_stuff}}
    greentea_send_kv("gimme_something", "some_stuff"); 

    char key[16] = {0};
    char value[32] = {0};
    // Blocking wait for master response for {{gimme_something; some_stuff}}
    greentea_parse_kv(key, value, sizeof(key), sizeof(value));
    // ...
    fprintf(stderr, "Received from master %s, %s", key, value);
    // ...

    GREENTEA_TSUITE_RESULT(true);    // __exit
}

Example of corresponding host test

class GimmeAuto(BaseHostTest):
    """ Simple, basic host test's test runner waiting for serial port
        output from MUT, no supervision over test running in MUT is executed.
    """

    __result = None
    name = "gimme_auto"

    def _callback_gimme_something(self, key, value, timestamp):
        # You've received {{gimme_something;*}}

        # We will send DUT some data back...
        # And now decide about test case result
        if value == 'some_stuff':
            # Message payload/value was 'some_stuff'
            # We can for example return true from test
            self.send_kv("print_this", "This is what I wanted %s"% value)
            self.notify_complete(True)
        else:
            self.send_kv("print_this", "This not what I wanted :(")
            self.notify_complete(False)

    def setup(self):
        # Register callback for message 'gimme_something' from DUT
        self.register_callback("gimme_something", self._callback_gimme_something)

        # Initialize your host test here
        # ...

    def result(self):
        # Define your test result here
        # Or use self.notify_complete(bool) to pass result anytime!
        return self.__result

    def teardown(self):
        # Release resources here after test is completed
        pass

Log:

[1454926794.22][HTST][INF] copy image onto target...
        1 file(s) copied.
[1454926801.48][HTST][INF] starting host test process...
[1454926802.01][CONN][INF] starting connection process...
[1454926802.01][CONN][INF] initializing serial port listener...
[1454926802.01][SERI][INF] serial(port=COM188, baudrate=9600)
[1454926802.02][SERI][INF] reset device using 'default' plugin...
[1454926802.27][SERI][INF] wait for it...
[1454926803.27][CONN][INF] sending preamble '9caa42a0-28a0-4b80-ba1d-befb4e43a4c1'...
[1454926803.27][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1454926803.27][SERI][TXD] {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}
[1454926803.40][CONN][RXD] {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}
[1454926803.40][CONN][INF] found SYNC in stream: {{__sync;9caa42a0-28a0-4b80-ba1d-befb4e43a4c1}}, queued...
[1454926803.40][HTST][INF] sync KV found, uuid=9caa42a0-28a0-4b80-ba1d-befb4e43a4c1, timestamp=1454926803.405000
[1454926803.42][CONN][RXD] {{__timeout;15}}
[1454926803.42][CONN][INF] found KV pair in stream: {{__timeout;15}}, queued...
[1454926803.42][HTST][INF] setting timeout to: 15 sec
[1454926803.45][CONN][RXD] {{__host_test_name;gimme_auto}}
[1454926803.45][CONN][INF] found KV pair in stream: {{__host_test_name;gimme_auto}}, queued...
[1454926803.45][HTST][INF] host test setup() call...
[1454926803.45][HTST][INF] CALLBACKs updated
[1454926803.45][HTST][INF] host test detected: gimme_auto
[1454926803.48][CONN][RXD] {{gimme_something;some_stuff}}
[1454926803.48][CONN][INF] found KV pair in stream: {{gimme_something;some_stuff}}, queued...
[1454926803.48][SERI][TXD] {{print_this;This is what I wanted some_stuff}}
[1454926803.48][HTST][INF] __notify_complete(True)
[1454926803.62][CONN][RXD] Received from master print_this, This is what I wanted some_stuf
[1454926803.62][CONN][RXD] {{end;success}}
[1454926803.62][CONN][INF] found KV pair in stream: {{end;success}}, queued...
[1454926803.62][HTST][ERR] orphan event in main phase: {{end;success}}, timestamp=1454926803.625000
[1454926803.63][CONN][RXD] {{__exit;0}}
[1454926803.63][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1454926803.63][HTST][INF] __exit(0)
[1454926803.63][HTST][INF] test suite run finished after 0.21 sec...
[1454926803.63][HTST][INF] exited with code: None
[1454926803.63][HTST][INF] 0 events in queue
[1454926803.63][HTST][INF] stopped consuming events
[1454926803.63][HTST][INF] host test result() skipped, received: True
[1454926803.63][HTST][INF] calling blocking teardown()
[1454926803.63][HTST][INF] teardown() finished
[1454926803.63][HTST][INF] {{result;success}}
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226d94b0e770000000000000000000000002492f3cf
mbedgt: test suite 'mbed-drivers-test-gimme' ......................................................... OK in 10.02 sec
mbedgt: shuffle seed: 0.3631708941
mbedgt: test suite report:
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| target        | platform_name | test suite              | result | elapsed_time (sec) | copy_method |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-gimme | OK     | 10.02              | shell       |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 OK

Host test examples

Return result after __exit

class GimmeAuto(BaseHostTest):
    """ Simple, basic host test's test runner waiting for serial port
        output from MUT, no supervision over test running in MUT is executed.
    """

    __result = None
    name = "gimme_auto"

    def _callback_gimme_something(self, key, value, timestamp):
        # You've received {{gimme_something;*}}

        # We will send DUT some data back...
        # And now decide about test case result
        if value == 'some_stuff':
            # Message payload/value was 'some_stuff'
            # We can for example return true from test
            self.send_kv("print_this", "This is what I wanted %s"% value)
            self.__result = True
        else:
            self.send_kv("print_this", "This not what I wanted :(")
            self.__result = False

    def setup(self):
        # Register callback for message 'gimme_something' from DUT
        self.register_callback("gimme_something", self._callback_gimme_something)

        # Initialize your host test here
        # ...

    def result(self):
        # Define your test result here
        # Or use self.notify_complete(bool) to pass result anytime!
        return self.__result

    def teardown(self):
        # Release resources here after test is completed
        pass

Corresponding log:

[1454926627.11][HTST][INF] copy image onto target...
        1 file(s) copied.
[1454926634.38][HTST][INF] starting host test process...
[1454926634.93][CONN][INF] starting connection process...
[1454926634.93][CONN][INF] initializing serial port listener...
[1454926634.93][SERI][INF] serial(port=COM188, baudrate=9600)
[1454926634.94][SERI][INF] reset device using 'default' plugin...
[1454926635.19][SERI][INF] wait for it...
[1454926636.19][CONN][INF] sending preamble '9a743ff3-45e6-44cf-9e2a-9a83e6205184'...
[1454926636.19][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1454926636.19][SERI][TXD] {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}
[1454926636.33][CONN][RXD] {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}
[1454926636.33][CONN][INF] found SYNC in stream: {{__sync;9a743ff3-45e6-44cf-9e2a-9a83e6205184}}, queued...
[1454926636.33][HTST][INF] sync KV found, uuid=9a743ff3-45e6-44cf-9e2a-9a83e6205184, timestamp=1454926636.331000
[1454926636.34][CONN][RXD] {{__timeout;15}}
[1454926636.34][CONN][INF] found KV pair in stream: {{__timeout;15}}, queued...
[1454926636.34][HTST][INF] setting timeout to: 15 sec
[1454926636.38][CONN][RXD] {{__host_test_name;gimme_auto}}
[1454926636.38][CONN][INF] found KV pair in stream: {{__host_test_name;gimme_auto}}, queued...
[1454926636.38][HTST][INF] host test setup() call...
[1454926636.38][HTST][INF] CALLBACKs updated
[1454926636.38][HTST][INF] host test detected: gimme_auto
[1454926636.41][CONN][RXD] {{gimme_something;some_stuff}}
[1454926636.41][CONN][INF] found KV pair in stream: {{gimme_something;some_stuff}}, queued...
[1454926636.41][SERI][TXD] {{print_this;This is what I wanted some_stuff}}
[1454926636.54][CONN][RXD] Received from master print_this, This is what I wanted some_stuf
[1454926636.54][CONN][RXD] {{end;success}}
[1454926636.54][CONN][INF] found KV pair in stream: {{end;success}}, queued...
[1454926636.55][HTST][ERR] orphan event in main phase: {{end;success}}, timestamp=1454926636.541000
[1454926636.56][CONN][RXD] {{__exit;0}}
[1454926636.56][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1454926636.56][HTST][INF] __exit(0)
[1454926636.56][HTST][INF] test suite run finished after 0.22 sec...
[1454926636.56][HTST][INF] exited with code: None
[1454926636.56][HTST][INF] 0 events in queue
[1454926636.56][HTST][INF] stopped consuming events
[1454926636.56][HTST][INF] host test result(): True
[1454926636.56][HTST][INF] calling blocking teardown()
[1454926636.56][HTST][INF] teardown() finished
[1454926636.56][HTST][INF] {{result;success}}
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226d94b0e770000000000000000000000002492f3cf
mbedgt: test suite 'mbed-drivers-test-gimme' ......................................................... OK in 10.04 sec
mbedgt: shuffle seed: 0.3866075474
mbedgt: test suite report:
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| target        | platform_name | test suite              | result | elapsed_time (sec) | copy_method |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-gimme | OK     | 10.04              | shell       |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 OK

TODO

• Code coverage feature support is not implemented in this version.

Changes

See corresponding changes in:

• mbed-drivers and
• htrun

Example use / test change

• Clone this PR, mbed-drivers and htrun.
• Prepare test environment:

$ cd htrun
[devel_transport] $ python setup.py develop
$ cd greentea
[devel_transport] $ python setup.py develop

• Build updated with new async tests inside my mbed-drivers:

$ cd mbed-drivers
[devel_greentea] $ yt target frdm-k64f-gcc
[devel_greentea] $ yt build

• Run greentea (skip build to avoid errors related to core-util dependencies mismatch):
  (you can start gently with one test case:)

$ mbedgt -V --skip-build -n mbed-drivers-test-basic

You should get something like this:

mbedgt: checking for yotta target in current directory
        reason: no --target switch set
mbedgt: checking yotta target in current directory
        calling yotta: yotta --plain target
mbedgt: assuming default target as 'frdm-k64f-gcc'
mbedgt: detecting connected mbed-enabled devices...
mbedgt: detected 1 device
        detected 'K64F' -> 'K64F[0]', console at 'COM188', mounted at 'E:', target id '02400226d94b0e770000000000000000000000002492f3cf'
mbedgt: local yotta target search in './yotta_targets' for compatible mbed-target 'k64f'
        inside './yotta_targets\frdm-k64f-gcc' found compatible target 'frdm-k64f-gcc'
mbedgt: processing 'frdm-k64f-gcc' yotta target compatible platforms...
mbedgt: processing 'K64F' platform...
mbedgt: using platform 'K64F' for test:
        target_id_mbed_htm = '02400226d94b0e770000000000000000000000002492f3cf'
        mount_point = 'E:'
        target_id = '02400226d94b0e770000000000000000000000002492f3cf'
        serial_port = 'COM188'
        target_id_usb_id = '02400226d94b0e770000000000000000000000002492f3cf'
        platform_name = 'K64F'
        platform_name_unique = 'K64F[0]'
mbedgt: skipping calling yotta (specified with --skip-build option)
mbedgt: test case filter (specified with -n option)
        test filtered in 'mbed-drivers-test-basic'
mbedgt: running 1 test for target 'frdm-k64f-gcc' and platform 'K64F'
        use 1 instance for testing
mbedgt: selecting test case observer...
        calling mbedhtrun: mbedhtrun -d E: -p COM188:9600 -f ".\build\frdm-k64f-gcc\test\mbed-drivers-test-basic.bin" -C 4 -c shell -m K64F
mbedgt: mbed-host-test-runner: started
[1454683943.05][HTST][INF] copy image onto target...
        1 file(s) copied.
[1454683950.35][HTST][INF] starting host test process...
[1454683950.93][CONN][INF] starting connection process...
[1454683950.93][CONN][INF] initializing serial port listener...
[1454683950.93][SERI][INF] serial(port=COM188, baudrate=9600)
[1454683950.93][SERI][INF] reset device using 'default' plugin...
[1454683951.18][SERI][INF] wait for it...
[1454683952.18][CONN][INF] sending preamble 'aa4c49dd-8539-4a97-8883-9841320adc16'...
[1454683952.18][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1454683952.18][SERI][TXD] {{__sync;aa4c49dd-8539-4a97-8883-9841320adc16}}
[1454683952.31][CONN][RXD] {{__sync;aa4c49dd-8539-4a97-8883-9841320adc16}}
[1454683952.31][CONN][INF] found SYNC in stream: {{__sync;aa4c49dd-8539-4a97-8883-9841320adc16}}, queued...
[1454683952.31][HTST][INF] sync KV found, uuid=aa4c49dd-8539-4a97-8883-9841320adc16, timestamp=1454683952.314000
[1454683952.32][CONN][RXD] {{__timeout;5}}
[1454683952.32][CONN][INF] found KV pair in stream: {{__timeout;5}}, queued...
[1454683952.32][HTST][INF] setting timeout to: 5 sec
[1454683952.38][CONN][RXD] {{__host_test_name;default_auto}}
[1454683952.38][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1454683952.38][HTST][INF] host test setup() call...
[1454683952.38][HTST][INF] CALLBACKs updated
[1454683952.38][HTST][INF] host test detected: default_auto
[1454683952.38][CONN][RXD] {{end;success}}
[1454683952.38][CONN][INF] found KV pair in stream: {{end;success}}, queued...
[1454683952.39][HTST][INF] __notify_complete(True)
[1454683952.40][CONN][RXD] {{__exit;0}}
[1454683952.40][CONN][INF] found KV pair in stream: {{__exit;0}}, queued...
[1454683952.40][HTST][INF] __exit(0)
[1454683952.40][HTST][INF] test suite run finished after 0.08 sec...
[1454683952.40][HTST][INF] exited with code: None
[1454683952.40][HTST][INF] 0 events in queue
[1454683952.40][HTST][INF] stopped consuming events
[1454683952.40][HTST][INF] host test result() skipped, received: True
[1454683952.40][HTST][INF] calling blocking teardown()
[1454683952.40][HTST][INF] teardown() finished
[1454683952.40][HTST][INF] greentea formatted output section...
{{success}}
{{end}}
mbedgt: mbed-host-test-runner: stopped
mbedgt: mbed-host-test-runner: returned 'OK'
mbedgt: test on hardware with target id: 02400226d94b0e770000000000000000000000002492f3cf
        test 'mbed-drivers-test-basic' ......................................................... OK in 10.00 sec
mbedgt: shuffle seed: 0.2393565981
mbedgt: test report:
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| target        | platform_name | test                    | result | elapsed_time (sec) | copy_method |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
| frdm-k64f-gcc | K64F          | mbed-drivers-test-basic | OK     | 10.0               | shell       |
+---------------+---------------+-------------------------+--------+--------------------+-------------+
mbedgt: results: 1 OK
mbedgt: completed in 11.20 sec

[0xc0170]

Still valid to use capital letter for this local variable?

[PrzemekWirkus]

I will change it to lower case !

[PrzemekWirkus]

PS: CircleCI dies because of not mocked stdout in Process/Popen.