[WIP] [TF:OTA] OTA updates common work branch

[kYc0o]

Contribution description

DO NOT MERGE! - PLEASE DON'T TRIGGER THE CI UNLESS IT'S REALLY NEEDED

This is a PR to track the work in progress for the OTA updates task force. Some PRs were already merged under different contexts, but they are essential to achieve this goal.

Basically, this PR represents what's described in #9342 as a "ready to test" environment, on which all the features are already (dirty) implemented and work as expected (no bug free warranties). The goal is to extract little by little the features and cleanly integrate them into RIOT, which is a lesson learned from the several previous attempts.

Testing procedure

Basically, we provide two applications (which might be PRed conveniently and might differ of what they are now) :

• examples/ota: simple push-based OTA update example. The README.md in there provides enough information to test it.
• examples/suit_updater: this application leverages the SUIT manifest format to perform updates from an internet server. It's quite complex and we don't have a README.md yet, but I'm working on it and will post it soon.

Both applications embed security prototypes to perform signature and validation of it with Ed25519, currently with the tweetnacl package.

Following what is described on examples/ota/README.md, the first step is to generate the keys with the following commands:

$ cd dist/tools/firmware
$ make
$ bin/firmware genkeys sec.key pub.key

To flash the device (currently tested only on samr21-xpro and iotlab-m3) with either ota or suit_updater:

• BOARD=samr21-xpro APP_VER=$(date +%s) make -C examples/ota riotboot/flash: flash bootloader plus application firmware on slot1 (initial state).
• BOARD=samr21-xpro APP_VER=$(date +%s) make -C examples/ota riotboot/slot2: generate an "update" of the same firmware, since the date is used as a version it will be always higher. The output file has the following format ota_example-slot2.signed.bin. That's the file that needs to be sent to the device via CoAP, e.g. for ota example. For suit_updater there are more steps, as the generation of the manifest requires more input data.

I hope with this some people can already play a bit.

Issues/PRs references

#9342 as the tracking issue
https://github.com/RIOT-OS/RIOT/projects/11 as the tracking project with the ongoing PRs, already merged PRs and superseded PRs.

Thanks (really) a lot to @kaspar030 @cladmi and @bergzand for their invaluable help to achieve this feature!

[kYc0o]

Ping to interested people:

@fedepell
@danpetry
@OYTIS
@emmanuelsearch

[fedepell]

@kYc0o : thanks for the ping!

I had just still one more problem compared to the wip. After this kYc0o#10 now it works very well! (did a few rebuilds and swaps between two slots)

Will try to test further from tomorrow (OOO today) and try to give more inputs.

Thanks for your work!

[kYc0o]

Thanks @fedepell for your interest and contributions! We'll review that asap.

I also forgot to ping @basilfx who was testing older versions of this.

[fedepell]

Just some more feedback from testing on my side:

I completed also the upgrade via CoAP via serial on a saml-xpro21 using ethos. Sorry for the late feedback but I had quite some problems but at the end unrelated to this issue (see issue #10068) but once that was solved now it works like a charm, so can switch also by transfering via CoAP (over ethos in my case).

I'll do a smal PR with some tiny doc fixes.

Question: would it be interesting for the patch to have a couple of lines in the readme on how you can test it via ethos as an additional example? If so I could prepare.

Just some output ;)

ota: received bytes 0-64
ota: initializing update to target slot 2
ota: received bytes 64-128
ota: received bytes 128-192
ota: received bytes 192-256
ota: verifying metadata ...
ota: verification successful
ota: received bytes 192-256 of 54620 (left=54364)
coap_ota_handler(): ignoring already received block
...
ota: received bytes 54592-54620 of 54620 (left=0)
coap_ota_handler(): ignoring already received block
ota: firmware flashing completed successfully
ota: firmware flashing completed successfully

[kYc0o]

Question: would it be interesting for the patch to have a couple of lines in the readme on how you can test it via ethos as an additional example? If so I could prepare.

Highly appreciated!! Thanks for testing!!

[kYc0o]

Rebased to current master. Removed commits related to coap block2 support and "fake" dependencies on stm32 devices for flashpage_write_common.

[fedepell]

@kYc0o: any plans on continuing this effort? I would be interested to continue and was wondering if this could align to current master (with the riotboot changes there) so I would not be using this branch.

[kYc0o]

@fedepell yes! It's in my plans to continue. I'd like #10215 to come first before I rebase this branch, since it would be less work. Let's try to merge #10215 soon so we can continue here.

[fedepell]

@fedepell yes! It's in my plans to continue. I'd like #10215 to come first before I rebase this branch, since it would be less work. Let's try to merge #10215 soon so we can continue here.

Ok, didn't notice that PR sorry (please if you remember just ping me when you have related work, I sometimes get lost ;) ) .
Will join testing/reviewing effort in the next days for sure, was looking forward to that and firmware tools! Thanks!

[kYc0o]

What's the status here? Several related PRs were merged and this got super outdated. Is of some use to still having it open?

[fedepell]

I agree, I believe mostly the OTA suit is still missing for this to be complete. I had a short chat at FOSDEM BOH with @bergzand , not sure if something continued after that (gave a search on github but didn't find anything).

[emmanuelsearch]

@kaspar030 @bergzand does it make more sense to close this indeed?
The plan is to consider bits of another branch such as this one right?

[emmanuelsearch]

@kYc0o let's close this in favor of the cluster of PRs around #11818
(but feel free to reopen if you feel it's necessary)