Add HIL testing #1297

SergioGasquez · 2024-03-15T09:58:57Z

This PR adds the foundations for HIL, It's still in an early phase and might be draft for a while or get merged and get some follow-up PRs.

See the workflow runs in https://github.com/esp-rs/esp-hal/actions/workflows/hil.yml

For instructions on how the testing works, see the hil-test readme

Current board support

Only C3, C6 and H2 are supported
C2 should be supported soon
Xtensa devices would require adding Xtensa support in semihosting and probe-rs
- See Support Xtensa (OpenOCD Semihosting) taiki-e/semihosting#9 and Xtensa semihosting support probe-rs/probe-rs#2301
- S3 is now supported! We still need to upstream changes and setup a VM with it, so it will probably be in another PR.

Current tests

Only aes, gpio, uart and spi_full_duplex are partially being tested. See full list of peripherals pending in: Set up hardware-in-loop testing #809 (comment)

Todo

Clear up fixmes and todos in code
Decide when the workflow needs to run
- Run when the PR is added to the merge_queue and abort the merge if the CI fails
- Add a way to manually trigger the workflow with a given repo/branch.
  - Couldn't test those two triggers. Merge queues might require changing some settings, see https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue#how-merge-queues-work and https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#merge_group
Improve documentation on the VMs and wiring in the devkits
- Atm, we are documenting it on the readme, should we use any other method?
Find a way to cfg away tests for test for targets.
- Initially though of required-features, but not really an option since we cant set up multiple chips.
- There is already a solution, not ideal but does the purpose. See Add HIL testing #1297 (comment)
Shall we keep the log and defmt features or use only one of them?
- Removed log feature and enabled defmt by default: 57e058b

Please, feel free to report any feedback or input that you have!

hil-test/src/lib.rs

hil-test/tests/aes.rs

hil-test/README.md

hil-test/tests/spi_full_duplex.rs

SergioGasquez · 2024-03-18T16:01:05Z

Just pushed 700a86b which shows a way to configure away tests of peripherals for targets that dont support that peripheral. Not an ideal solution imo, but couldn't find any other way (thanks @t-moe for the suggestion)

hil-test/.cargo/config.toml

jessebraham

I think, especially for our first iteration, this is looking really good! Thank you so much for all your work here, this is a huge win for us.

We should probably expand the section in README.md regarding adding tests, but that is absolutely beyond this scope of this PR and can be tackled separately.

So I'll say LGTM, but let's wait for confirmation from @MabezDev and/or @bjoernQ as well :)

jessebraham · 2024-03-20T16:11:45Z

Also I guess we will need to wait until this is merged until we can add the relevant jobs to the required merge queue checks, so HIL will probably not run when this is merged (but will for the next PR)?

Just kidding I see them, I guess because the branch belongs to this repo. So I will add them as soon as I'm ready to merge!

SergioGasquez · 2024-03-21T09:31:05Z

Sorry for adding late changes.

I stole the alias idea and implemented it here to have a simple way of running all the tests for a target
Pinned the moving dependencies (semihosting, embedded-test and probe-rs)
Updated the README

After the pinning/alias changes, I ran the CI to verify that everything was working, here is the run https://github.com/esp-rs/esp-hal/actions/runs/8372136967 (I amended the commit and removed the push trigger of the workflow afterward)

bjoernQ

LGTM ❤️

MabezDev

Thank you @SergioGasquez!

* feat: Update probe-rs/embedded-test to probe-rs#2292 * feat: Remove lib

SergioGasquez · 2024-03-21T14:25:33Z

Just tested all the targets locally:

ESP32C3:

Results

❯ cargo esp32c3
warning: method `into_result` is never used
   --> /home/sergio/Documents/Espressif/esp-rs/esp-hal/esp-hal/src/fmt.rs:204:8
    |
201 | pub trait Try {
    |           --- method in this trait
...
204 |     fn into_result(self) -> Result<Self::Ok, Self::Error>;
    |        ^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: `esp-hal` (lib) generated 1 warning
    Finished `release` profile [optimized + debuginfo] target(s) in 0.06s
     Running tests/aes.rs (target/riscv32imc-unknown-none-elf/release/deps/aes-40fe1a1875ed0f4c)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 239.08 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.52 KiB/35.52 KiB @ 64.83 KiB/s (eta 0s )    Finished in 1.363s

running 2 tests
test tests::test_aes_encryption ... ok
test tests::test_aes_decryption ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.87s

     Running tests/gpio.rs (target/riscv32imc-unknown-none-elf/release/deps/gpio-3ace83cd88ceff7c)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 238.35 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.26 KiB/35.26 KiB @ 64.59 KiB/s (eta 0s )    Finished in 1.363s

running 2 tests
test tests::test_gpio_input  ... ok
test tests::test_gpio_output ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.86s

     Running tests/spi_full_duplex.rs (target/riscv32imc-unknown-none-elf/release/deps/spi_full_duplex-28d01ab612584813)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 240.25 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 37.06 KiB/37.06 KiB @ 66.70 KiB/s (eta 0s )    Finished in 1.367s

running 4 tests
test tests::test_symestric_transfer                      ... ok
test tests::test_asymestric_transfer                     ... ok
test tests::test_symestric_transfer_huge_buffer          ... ok
test tests::test_symestric_transfer_huge_buffer_no_alloc ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.72s

     Running tests/uart.rs (target/riscv32imc-unknown-none-elf/release/deps/uart-12ebab303e43539b)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 236.60 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.19 KiB/35.19 KiB @ 64.28 KiB/s (eta 0s )    Finished in 1.37s

running 1 test
test tests::test_send_receive ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.44s

ESP32-C6:

Results

   Compiling hil-test v0.0.0 (/home/sergio/Documents/Espressif/esp-rs/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 6.21s
     Running tests/aes.rs (target/riscv32imac-unknown-none-elf/release/deps/aes-79d2fe8d6b7de728)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 128.00 KiB/128.00 KiB @ 634.74 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 36.33 KiB/36.33 KiB @ 96.69 KiB/s (eta 0s )    Finished in 0.589s

running 2 tests
test tests::test_aes_encryption ... ok
test tests::test_aes_decryption ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.86s

     Running tests/gpio.rs (target/riscv32imac-unknown-none-elf/release/deps/gpio-92410ef5e9f8728d)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 128.00 KiB/128.00 KiB @ 646.24 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 36.06 KiB/36.06 KiB @ 96.54 KiB/s (eta 0s )    Finished in 0.583s

running 2 tests
test tests::test_gpio_input  ... ok
test tests::test_gpio_output ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.86s

     Running tests/spi_full_duplex.rs (target/riscv32imac-unknown-none-elf/release/deps/spi_full_duplex-5f2314b91cd414c3)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 128.00 KiB/128.00 KiB @ 649.48 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 37.98 KiB/37.98 KiB @ 97.06 KiB/s (eta 0s )    Finished in 0.6s

running 4 tests
test tests::test_symestric_transfer                      ... ok
test tests::test_asymestric_transfer                     ... ok
test tests::test_symestric_transfer_huge_buffer          ... ok
test tests::test_symestric_transfer_huge_buffer_no_alloc ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.71s

     Running tests/uart.rs (target/riscv32imac-unknown-none-elf/release/deps/uart-456906ed79152f2a)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 128.00 KiB/128.00 KiB @ 653.25 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 36.02 KiB/36.02 KiB @ 96.73 KiB/s (eta 0s )    Finished in 0.579s

running 1 test
test tests::test_send_receive ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s

ESP32-H2:

Results

   Compiling hil-test v0.0.0 (/home/sergio/Documents/Espressif/esp-rs/esp-hal/hil-test)
    Finished `release` profile [optimized + debuginfo] target(s) in 9.01s
     Running tests/aes.rs (target/riscv32imac-unknown-none-elf/release/deps/aes-de854d1506d0a923)
      Erasing ✔ [00:00:00] [#######################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 1.03 MiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.34 KiB/35.34 KiB @ 67.38 KiB/s (eta 0s )    Finished in 0.718s

running 2 tests
test tests::test_aes_encryption ... ok
test tests::test_aes_decryption ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.86s

     Running tests/gpio.rs (target/riscv32imac-unknown-none-elf/release/deps/gpio-46b476d876631f56)
      Erasing ✔ [00:00:00] [####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 1018.32 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.06 KiB/35.06 KiB @ 66.54 KiB/s (eta 0s )    Finished in 0.727s

running 2 tests
test tests::test_gpio_input  ... ok
test tests::test_gpio_output ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.86s

     Running tests/spi_full_duplex.rs (target/riscv32imac-unknown-none-elf/release/deps/spi_full_duplex-2b0143ea09f9469e)
      Erasing ✔ [00:00:00] [#######################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 1.00 MiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 37.05 KiB/37.05 KiB @ 66.30 KiB/s (eta 0s )    Finished in 0.758s

running 4 tests
test tests::test_symestric_transfer                      ... ok
test tests::test_asymestric_transfer                     ... ok
test tests::test_symestric_transfer_huge_buffer          ... ok
test tests::test_symestric_transfer_huge_buffer_no_alloc ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.71s

     Running tests/uart.rs (target/riscv32imac-unknown-none-elf/release/deps/uart-1ce1290e074c134a)
      Erasing ✔ [00:00:00] [#######################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 1.01 MiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 35.01 KiB/35.01 KiB @ 66.46 KiB/s (eta 0s )    Finished in 0.723s

running 1 test
test tests::test_send_receive ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.43s

ESP32-S3:

Results

   Compiling hil-test v0.0.0 (/home/sergio/Documents/Espressif/esp-rs/esp-hal/hil-test)
    Finished release [optimized + debuginfo] target(s) in 18.56s
     Running tests/aes.rs (target/xtensa-esp32s3-none-elf/release/deps/aes-e68628e057fe0ffb)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 336.62 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 39.93 KiB/39.93 KiB @ 58.32 KiB/s (eta 0s )    Finished in 1.268s

running 2 tests
test tests::test_aes_encryption ... ok
test tests::test_aes_decryption ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.84s

     Running tests/gpio.rs (target/xtensa-esp32s3-none-elf/release/deps/gpio-e3b788197b823785)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 333.05 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 39.64 KiB/39.64 KiB @ 58.17 KiB/s (eta 0s )    Finished in 1.27s

running 2 tests
test tests::test_gpio_input  ... ok
test tests::test_gpio_output ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.85s

     Running tests/spi_full_duplex.rs (target/xtensa-esp32s3-none-elf/release/deps/spi_full_duplex-737c4179a9571d7f)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 333.78 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 41.73 KiB/41.73 KiB @ 59.56 KiB/s (eta 0s )    Finished in 1.289s

running 4 tests
test tests::test_symestric_transfer                      ... ok
test tests::test_asymestric_transfer                     ... ok
test tests::test_symestric_transfer_huge_buffer          ... ok
test tests::test_symestric_transfer_huge_buffer_no_alloc ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.69s

     Running tests/uart.rs (target/xtensa-esp32s3-none-elf/release/deps/uart-0c3c6e9f8846253d)
      Erasing ✔ [00:00:00] [#####################################################################################################################################################################################] 192.00 KiB/192.00 KiB @ 331.86 KiB/s (eta 0s )
  Programming ✔ [00:00:00] [########################################################################################################################################################################################] 39.71 KiB/39.71 KiB @ 58.35 KiB/s (eta 0s )    Finished in 1.272s

running 1 test
test tests::test_send_receive ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.42s

* Create the `hil-test` package * Add a simple integration test to verify basic GPIO functionality * WIP * feat: Update with esp-hal unification * build: Update dependencies * feat: Add a simple CI workflow test * ci: Avoid using a gh-hosted-runner to build * ci: Remove building bins in gh-hosted-runner * ci: Remove HIL Gpio CI test * ci: Test all the available tests * test: Add spi_full_duplex test * docs: Add documentation * test: Add uart test * style: Remove unused imports * docs: Update wiring, document H2 VM * ci: Enable H2 tests * ci: Add rust-cache action * docs: Document H2 vm * test: Add timeout * ci: Enable ESP32-C3 tests * feat: Add timeouts * feat: Add aes test * ci: Avoid running CI workflow when we change hil-test stuff * test: Remove warnings * feat: Address feedback * feat: Update features names and spi methods * ci: Remove rust-cache action * Update HIL to probe-rs#2292 (esp-rs#1307) * feat: Update probe-rs/embedded-test to probe-rs#2292 * feat: Remove lib * ci: Use a matrix * ci: Enable ESP32C3 * feat: Add a way to cfg away test for unsuported peripherals * ci: Update trigger conditions * feat: Update pins to make it work on s3 * feat: Changes enabling S3 * feat: Remove log feature * feat: Adapt for rebase * feat: Remove env * feat: enable S3 * chore: Remove todo * build: Pin dependencies * feat: Add target alias * docs: Update readme * fix: Fix traits imports after rebase. Use debug * build: Remove lto * feat: Build tests on release mode --------- Co-authored-by: Jesse Braham <[email protected]>

SergioGasquez mentioned this pull request Mar 15, 2024

Set up hardware-in-loop testing #809

Closed

MabezDev requested changes Mar 15, 2024

View reviewed changes

hil-test/src/lib.rs Outdated Show resolved Hide resolved

hil-test/tests/aes.rs Show resolved Hide resolved

hil-test/README.md Outdated Show resolved Hide resolved

hil-test/tests/spi_full_duplex.rs Outdated Show resolved Hide resolved

SergioGasquez force-pushed the feat/hil-testing branch from 2327179 to 0347722 Compare March 15, 2024 11:36

SergioGasquez added the skip-changelog No changelog modification needed label Mar 15, 2024

SergioGasquez mentioned this pull request Mar 15, 2024

Support for Xtensa architecture taiki-e/semihosting#8

Closed

SergioGasquez force-pushed the feat/hil-testing branch from c82bce8 to f713d01 Compare March 18, 2024 11:27

jessebraham reviewed Mar 20, 2024

View reviewed changes

hil-test/.cargo/config.toml Outdated Show resolved Hide resolved

hil-test/.cargo/config.toml Outdated Show resolved Hide resolved

SergioGasquez force-pushed the feat/hil-testing branch from 04e8166 to 57e058b Compare March 20, 2024 15:04

SergioGasquez requested review from jessebraham and MabezDev March 20, 2024 15:52

SergioGasquez marked this pull request as ready for review March 20, 2024 16:01

SergioGasquez force-pushed the feat/hil-testing branch from fead731 to f895bf3 Compare March 20, 2024 16:01

jessebraham approved these changes Mar 20, 2024

View reviewed changes

SergioGasquez force-pushed the feat/hil-testing branch 5 times, most recently from 593a0cc to 1e6c16f Compare March 21, 2024 09:26

SergioGasquez force-pushed the feat/hil-testing branch from 7be1263 to ddd4f66 Compare March 21, 2024 09:31

bjoernQ approved these changes Mar 21, 2024

View reviewed changes

jessebraham enabled auto-merge March 21, 2024 13:11

MabezDev approved these changes Mar 21, 2024

View reviewed changes

jessebraham added this pull request to the merge queue Mar 21, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 21, 2024

SergioGasquez force-pushed the feat/hil-testing branch from ddd4f66 to 6f5bc9d Compare March 21, 2024 13:23

jessebraham added 2 commits March 21, 2024 15:07

Create the hil-test package

65ef75a

Add a simple integration test to verify basic GPIO functionality

ef6cdfb

SergioGasquez added 19 commits March 21, 2024 15:07

feat: Address feedback

6dbf0b2

feat: Update features names and spi methods

93447e2

ci: Remove rust-cache action

942c166

Update HIL to probe-rs#2292 (#1307)

1bc2a9a

* feat: Update probe-rs/embedded-test to probe-rs#2292 * feat: Remove lib

ci: Use a matrix

9df87f8

ci: Enable ESP32C3

6044ae6

feat: Add a way to cfg away test for unsuported peripherals

a1aa21d

ci: Update trigger conditions

77affa9

feat: Update pins to make it work on s3

72e214a

feat: Changes enabling S3

13fd9ea

feat: Remove log feature

a4202a1

feat: Adapt for rebase

c7c7dea

feat: Remove env

b94099c

feat: enable S3

9787fd1

chore: Remove todo

b736aef

build: Pin dependencies

17e3ca0

feat: Add target alias

43aeb6f

docs: Update readme

a9b6b8a

fix: Fix traits imports after rebase. Use debug

9849317

SergioGasquez force-pushed the feat/hil-testing branch from df2758b to 9849317 Compare March 21, 2024 14:07

SergioGasquez added 2 commits March 21, 2024 15:20

build: Remove lto

1c1d6a3

feat: Build tests on release mode

6020e8a

jessebraham enabled auto-merge March 21, 2024 14:27

jessebraham added this pull request to the merge queue Mar 21, 2024

Merged via the queue into main with commit baea915 Mar 21, 2024
17 checks passed

SergioGasquez deleted the feat/hil-testing branch March 21, 2024 14:55

SergioGasquez mentioned this pull request Apr 11, 2024

Add HIL tests for all peripherals which are practical to test #1334

Closed

38 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HIL testing #1297

Add HIL testing #1297

SergioGasquez commented Mar 15, 2024 •

edited

Loading

SergioGasquez commented Mar 18, 2024

jessebraham left a comment

jessebraham commented Mar 20, 2024 •

edited

Loading

SergioGasquez commented Mar 21, 2024

bjoernQ left a comment

MabezDev left a comment

SergioGasquez commented Mar 21, 2024 •

edited

Loading

Add HIL testing #1297

Add HIL testing #1297

Conversation

SergioGasquez commented Mar 15, 2024 • edited Loading

Current board support

Current tests

Todo

SergioGasquez commented Mar 18, 2024

jessebraham left a comment

Choose a reason for hiding this comment

jessebraham commented Mar 20, 2024 • edited Loading

SergioGasquez commented Mar 21, 2024

bjoernQ left a comment

Choose a reason for hiding this comment

MabezDev left a comment

Choose a reason for hiding this comment

SergioGasquez commented Mar 21, 2024 • edited Loading

SergioGasquez commented Mar 15, 2024 •

edited

Loading

jessebraham commented Mar 20, 2024 •

edited

Loading

SergioGasquez commented Mar 21, 2024 •

edited

Loading