Port mbed TLS self test to greentea #2603

andresag01 · 2016-09-01T17:00:31Z

This set of tests is a port of the mbed TLS test program at mbedtls/programs/test/selftest.c. The test executes the mbedtls_*_self_test() functions in the library modules and checks the output (PASS, FAIL). This patch ports the self test program to use mbedgt instead but maintains the same functionality.

andresag01 · 2016-09-01T17:02:11Z

@sbutcher-arm @yanesca @pjbakker: This patch addresses IOTSSL-941.

bridadan · 2016-09-07T19:53:57Z

@andresag01 Sorry for not looking at this sooner! It looks good, but the entropy test fails for targets that don't have the MBEDTLS_ENTROPY_HARDWARE_ALT macro set. Is this expected behavior? If so, could you add one more check for this macro before enabling the entropy test?

andresag01 · 2016-09-08T08:36:42Z

@bridadan thanks for having a look at this pull request. Could you please let us know what are the targets that fail? I presume that the reason for the failure is because they do not have entropy source enabled, which part of what we wanted to test. I am not sure what is the intended behaviour in this case, what do @yanesca, @sbutcher-arm and @pjbakker think?

screamerbg · 2016-09-08T11:12:43Z

@adustm Please let me know if you're able to run the tests in this PR against the HW entropy for STM32 that you recently implemented.

bridadan · 2016-09-08T17:04:46Z

@andresag01 I think that's probably the case, though I admit I'm not all that familiar with mbedTLS and the entropy sources.

If that test cannot pass on targets that don't support the entropy source yet, then I would think it doesn't make sense to run the test since there's no way it will pass. Instead it should be ran only when the feature is enabled.

sg- · 2016-09-09T21:13:26Z

The test could be gated by the macro as others are

#if !MBEDTLS_ENTROPY_HARDWARE_ALT
    #error [NOT_SUPPORTED] IPV4 not supported for this target
#endif

adustm · 2016-09-12T12:34:10Z

Hello,
I agree with @-sg, you should add the error reporting if the target does not support the mbedTLS entropy.
However I have tested it with one of ST boards that has the HW_entropy enabled. It works fine.
Here is the outcome for NUCLEO_F746ZG :

mbed test -m NUCLEO_F746ZG -t GCC_ARM -n tests-mbedtls-selftest
Build successes:
  * NUCLEO_F746ZG::GCC_ARM::MBED-BUILD
  * NUCLEO_F746ZG::GCC_ARM::TESTS-MBEDTLS-SELFTEST
mbedgt: greentea test automation tool ver. 1.2.0
mbedgt: test specification file 'C:\MCDGitW601\mbed\.build/tests\NUCLEO_F746ZG\GCC_ARM\test_spec.json' (specified with --test-spec option)
mbedgt: using 'C:\MCDGitW601\mbed\.build/tests\NUCLEO_F746ZG\GCC_ARM\test_spec.json' from current directory!
mbedgt: detecting connected mbed-enabled devices...
mbedgt: detected 1 device
mbedgt: processing target 'NUCLEO_F746ZG' toolchain 'GCC_ARM' compatible platforms... (note: switch set to --parallel 1)
mbedgt: test case filter (specified with -n option)
        test filtered in 'tests-mbedtls-selftest'
mbedgt: running 1 test for platform 'NUCLEO_F746ZG' and toolchain 'GCC_ARM'
mbedgt: mbed-host-test-runner: started
mbedgt: checking for GCOV data...
mbedgt: test on hardware with target id: 08160221054560033A01F928
mbedgt: test suite 'tests-mbedtls-selftest' .......................................................... OK in 17.30 sec
        test case: 'mbedtls_aes_self_test' ........................................................... OK in 0.49 sec
        test case: 'mbedtls_base64_self_test' ........................................................ OK in 0.05 sec
        test case: 'mbedtls_ccm_self_test' ........................................................... OK in 0.05 sec
        test case: 'mbedtls_ctr_drbg_self_test' ...................................................... OK in 0.05 sec
        test case: 'mbedtls_ecp_self_test' ........................................................... OK in 1.88 sec
        test case: 'mbedtls_entropy_self_test' ....................................................... OK in 0.06 sec
        test case: 'mbedtls_gcm_self_test' ........................................................... OK in 0.05 sec
        test case: 'mbedtls_hmac_drbg_self_test' ..................................................... OK in 0.05 sec
        test case: 'mbedtls_mpi_self_test' ........................................................... OK in 0.07 sec
        test case: 'mbedtls_rsa_self_test' ........................................................... OK in 0.14 sec
        test case: 'mbedtls_sha256_self_test' ........................................................ OK in 0.30 sec
        test case: 'mbedtls_sha512_self_test' ........................................................ OK in 0.93 sec
        test case: 'mbedtls_x509_self_test' .......................................................... OK in 0.05 sec
mbedgt: test case summary: 13 passes, 0 failures
mbedgt: all tests finished!
mbedgt: shuffle seed: 0.5399348762
mbedgt: test suite report:
+-----------------------+---------------+------------------------+--------+--------------------+-------------+
| target                | platform_name | test suite             | result | elapsed_time (sec) | copy_method |
+-----------------------+---------------+------------------------+--------+--------------------+-------------+
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | OK     | 17.3               | shell       |
+-----------------------+---------------+------------------------+--------+--------------------+-------------+
mbedgt: test suite results: 1 OK
mbedgt: test case report:
+-----------------------+---------------+------------------------+-----------------------------+--------+--------+--------+--------------------+
| target                | platform_name | test suite             | test case                   | passed | failed | result | elapsed_time (sec) |
+-----------------------+---------------+------------------------+-----------------------------+--------+--------+--------+--------------------+
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_aes_self_test       | 1      | 0      | OK     | 0.49               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_base64_self_test    | 1      | 0      | OK     | 0.05               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_ccm_self_test       | 1      | 0      | OK     | 0.05               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_ctr_drbg_self_test  | 1      | 0      | OK     | 0.05               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_ecp_self_test       | 1      | 0      | OK     | 1.88               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_entropy_self_test   | 1      | 0      | OK     | 0.06               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_gcm_self_test       | 1      | 0      | OK     | 0.05               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_hmac_drbg_self_test | 1      | 0      | OK     | 0.05               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_mpi_self_test       | 1      | 0      | OK     | 0.07               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_rsa_self_test       | 1      | 0      | OK     | 0.14               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_sha256_self_test    | 1      | 0      | OK     | 0.3                |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_sha512_self_test    | 1      | 0      | OK     | 0.93               |
| NUCLEO_F746ZG-GCC_ARM | NUCLEO_F746ZG | tests-mbedtls-selftest | mbedtls_x509_self_test      | 1      | 0      | OK     | 0.05               |
+-----------------------+---------------+------------------------+-----------------------------+--------+--------+--------+--------------------+
mbedgt: test case results: 13 OK
mbedgt: completed in 17.45 sec

LGTM

Cheers
Armelle

bridadan · 2016-09-12T16:42:59Z

@sg- It should be noted that the the NOT_SUPPORTED functionality only works at compile time. So if you want the other tests to be built/ran even if the the hardware entropy source is not present, these tests will need to be moved into a different test file.

For example, TESTS/mbedtls/selftest/selftest.cpp for all tests that don't have a dependency on the hardware entropy source, and TESTS/mbedtls/hardware_entropy/hardware_entropy.cpp for tests that DO have a dependency on the hardware entropy source. The latter file would have the following check at the top of the code:

#if !MBEDTLS_ENTROPY_HARDWARE_ALT
    #error [NOT_SUPPORTED] Hardware entropy not supported for this target
#endif

bridadan · 2016-09-12T19:27:03Z

@andresag01 Probably should have tagged you in the above message too 😄

adustm · 2016-09-16T13:40:17Z

Hello, is there any news on the topic ?

andresag01 · 2016-09-16T13:49:11Z

We are implementing @sbutcher-arm proposal i.e. have an additional configuration file in mbed TLS that gets included when DEVICE_ENTROPY_SOURCE is not defined in targets.json. We have prepared two PRs for this:

Mbed-TLS/mbedtls#615 (mbed TLS)
#2716 (mbed OS)

They are still marked as DRAFT because this is still work in progress.

yanesca · 2016-09-27T12:29:13Z

+1

simonbutcher · 2016-09-27T13:18:37Z

The pull request looks good to me.

0xc0170 · 2016-09-28T15:04:55Z

/morph test

0xc0170

+1

mbed-bot · 2016-09-28T15:27:27Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 980

Build failed!

bridadan · 2016-09-28T16:22:29Z

@0xc0170 Looks like a few targets can't fit this test in memory. Perhaps we need to break this up into a few binaries?

0xc0170 · 2016-09-28T16:29:16Z

@0xc0170 Looks like a few targets can't fit this test in memory. Perhaps we need to break this up into a few binaries?

@andresag01 Can you please have a look at failures, and how we can fix those?

0xc0170 · 2016-09-29T10:23:31Z

We had a quick look, the small platforms are having problems. RAM overflows, by >1k and similar sizes, therefore might be considered if those targets are either supported or should be (memory size restrictions).

simonbutcher · 2016-09-29T16:03:44Z

Because mbed TLS is a static link library you only need to build in the functions that you need, and you can leave out everything else. If you only want AES - you can link in only AES and exclude all hashing functions, X.509 support and everything else. Also, despite the fact asymmetric crypto may take a very long time on the smaller devices there may be some use cases that still want to do it if they don't care about the time taken to execute.

That means that's it's not obvious whether mbed TLS is appropriate or not for some of these tiny devices.

Therefore it still makes sense to run all tests on all devices for the moment as we can't second guess all use cases, even if some don't seem to make immediate sense.

Therefore, we'll break up the tests into smaller functional entities as @bridadan suggests to allow all tests to run. In this process we might find some of the bigger features still don't fit on the tiniest devices (such as X.509). If this is true we may look again at the feature set and see what makes sense and how to exclude such features.

bridadan · 2016-09-29T16:46:45Z

@sbutcher-arm I think that's probably the simplest approach for now. The one thing I would recommend is to try pairing some of the lightweight and related functions in the same test binary instead of creating one binary for each function. The reason being is flashing the binary takes the majority of the time for most of theses tests, so the testing time will increase quite a bit if you add 20 binaries.

andresag01 · 2016-09-29T16:56:04Z

@sbutcher-arm, @yanesca: I have rebased the changes and split the selftests into four functional components as previously discussed, but this is still failing the build test in some platforms. We could split these into even smaller parts.

bridadan · 2016-09-29T19:07:05Z

/morph test

mbed-bot · 2016-09-29T19:32:11Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 1005

Build failed!

bridadan · 2016-09-29T20:12:43Z

@andresag01 I know you said some platforms are failing, just reran the CI to get a feel for what is still affected.

simonbutcher · 2016-09-30T10:51:20Z

Examining the problem, it looks as if some tests (and also functions and features) won't fit on some devices without target specific optimisations. That means that we can't adopt a 'on configuration of tests will suit all targets' approach, and we'll need to find some way to configure tests per target - which we can't do today.

Because the immediate need is to test entropy sources, and for the sake of getting into the test framework some tests we can build on, we've removed all self-tests apart from entropy and SHA-2, which hopefully should be small and simple enough to run on all devices. From there we can expand the set of tests in later PR's.

0xc0170 · 2016-09-30T14:14:08Z

/morph test

mbed-bot · 2016-09-30T15:28:33Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 1011

All builds and test passed!

andresag01 changed the title ~~Port mbed TLS self test to greentea~~ DRAFT: Port mbed TLS self test to greentea Sep 2, 2016

andresag01 changed the title ~~DRAFT: Port mbed TLS self test to greentea~~ Port mbed TLS self test to greentea Sep 2, 2016

sg- mentioned this pull request Sep 8, 2016

HEXIWEAR: Enable the entropy collector for use in mbed TLS #2526

Merged

bogdanm added the needs: clairification label Sep 9, 2016

sg- mentioned this pull request Sep 9, 2016

STM32 hw entropy #2611

Merged

sg- added the needs: work label Sep 9, 2016

sg- removed the needs: clairification label Sep 17, 2016

sg- added needs: CI and removed needs: work labels Sep 27, 2016

andresag01 mentioned this pull request Sep 28, 2016

Update feature/mbedtls with mbedtls' development HEAD #2843

Merged

andresag01 force-pushed the mbedtls_self_test branch from be798c6 to 8e4faa0 Compare September 28, 2016 14:24

0xc0170 approved these changes Sep 28, 2016

View reviewed changes

andresag01 force-pushed the mbedtls_self_test branch from 8e4faa0 to 5fc271b Compare September 29, 2016 16:49

Port mbedTLS entropy and sha selftest to mbedOS

d6f40b6

andresag01 force-pushed the mbedtls_self_test branch from 5fc271b to d6f40b6 Compare September 30, 2016 10:25

sg- added ready for merge and removed needs: CI labels Sep 30, 2016

sg- merged commit 4c9ccd0 into ARMmbed:master Sep 30, 2016

sg- added release-version: 5.2.0 and removed ready for merge labels Sep 30, 2016

adbridge mentioned this pull request Mar 24, 2020

QSPI driver double call qspi_init() #12678

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port mbed TLS self test to greentea #2603

Port mbed TLS self test to greentea #2603

andresag01 commented Sep 1, 2016 •

edited

Loading

andresag01 commented Sep 1, 2016

bridadan commented Sep 7, 2016

andresag01 commented Sep 8, 2016

screamerbg commented Sep 8, 2016

bridadan commented Sep 8, 2016

sg- commented Sep 9, 2016

adustm commented Sep 12, 2016

bridadan commented Sep 12, 2016

bridadan commented Sep 12, 2016

adustm commented Sep 16, 2016

andresag01 commented Sep 16, 2016 •

edited

Loading

yanesca commented Sep 27, 2016

simonbutcher commented Sep 27, 2016

0xc0170 commented Sep 28, 2016

0xc0170 left a comment

mbed-bot commented Sep 28, 2016

bridadan commented Sep 28, 2016

0xc0170 commented Sep 28, 2016

0xc0170 commented Sep 29, 2016

simonbutcher commented Sep 29, 2016

bridadan commented Sep 29, 2016

andresag01 commented Sep 29, 2016 •

edited

Loading

bridadan commented Sep 29, 2016

mbed-bot commented Sep 29, 2016

bridadan commented Sep 29, 2016

simonbutcher commented Sep 30, 2016

0xc0170 commented Sep 30, 2016

mbed-bot commented Sep 30, 2016

Port mbed TLS self test to greentea #2603

Port mbed TLS self test to greentea #2603

Conversation

andresag01 commented Sep 1, 2016 • edited Loading

andresag01 commented Sep 1, 2016

bridadan commented Sep 7, 2016

andresag01 commented Sep 8, 2016

screamerbg commented Sep 8, 2016

bridadan commented Sep 8, 2016

sg- commented Sep 9, 2016

adustm commented Sep 12, 2016

bridadan commented Sep 12, 2016

bridadan commented Sep 12, 2016

adustm commented Sep 16, 2016

andresag01 commented Sep 16, 2016 • edited Loading

yanesca commented Sep 27, 2016

simonbutcher commented Sep 27, 2016

0xc0170 commented Sep 28, 2016

0xc0170 left a comment

Choose a reason for hiding this comment

mbed-bot commented Sep 28, 2016

Result: FAILURE

Output

bridadan commented Sep 28, 2016

0xc0170 commented Sep 28, 2016

0xc0170 commented Sep 29, 2016

simonbutcher commented Sep 29, 2016

bridadan commented Sep 29, 2016

andresag01 commented Sep 29, 2016 • edited Loading

bridadan commented Sep 29, 2016

mbed-bot commented Sep 29, 2016

Result: FAILURE

Output

bridadan commented Sep 29, 2016

simonbutcher commented Sep 30, 2016

0xc0170 commented Sep 30, 2016

mbed-bot commented Sep 30, 2016

Result: SUCCESS

Output

andresag01 commented Sep 1, 2016 •

edited

Loading

andresag01 commented Sep 16, 2016 •

edited

Loading

andresag01 commented Sep 29, 2016 •

edited

Loading