Add FOTA support and enable MCUboot logs

[chrisc11]

Summary

Get's a FOTA MVP up and running so we can get the latest and greatest on our nRF91 Thingy:91s

Change Highlights -- more details inline in the code:

1. I was going insane trying to debug MCUboot when no logs were being emitted. Tinkered around with Kconfig options to get this working. Trade off is I disabled support for mcumgr serial DFU but I do not believe anyone is using this atm anyway.
2. Fixing 1, pointed out that due to the size of the thingy91 image we are hitting an edge case with MCUboot's "swap move" algorithm blocking a FOTA from completing succesfully. I switched to OVERWRITE strategy to work around this.
3. First pass at a FOTA check. The algo works as follows:
   • Check for FOTA upon first connection since boot
   • Check for FOTA every 12 hours there-after - If FOTA check on 12 hour mark fails, check upon next successful connection (to accomodate situation where we were not connected)
4. There's some kind of corruption taking place when sending data to cloud and doing a FOTA. We should figure out what's going wrong but for now I fixed it by adding retry logic in our own custom FOTA handler (CONFIG_MEMFAULT_FOTA_DOWNLOAD_CALLBACK_CUSTOM=y)

Internal Documentation

https://www.notion.so/memfault/WeFault-Thingy91-Fleet-Quickstart-394ab5bd97ab4a49af86f40423f51982

Test Plan

1. Build the new image

west build -b [email protected]  memfault-asset-tracker -- -DCONFIG_MEMFAULT_NCS_PROJECT_KEY=\"${PROJECT_KEY}\" -DOVERLAY_CONFIG=overlay-memfault.conf -DCONFIG_BUILD_OUTPUT_META=n -Dmcuboot_CONFIG_BUILD_OUTPUT_META=n -DCONFIG_MEMFAULT_FOTA=y -DCONFIG_MEMFAULT_NCS_FW_VERSION_PREFIX=\"0.0.2+\"

2. Flash the image on the board

west flash --erase

3. Re-run 1) but with `-DCONFIG_MEMFAULT_NCS_FW_VERSION_PREFIX="0.0.3+"

4. Upload app_update.bin from build/zephyr folder into wefault project: https://app.memfault.com/organizations/memfault/projects/thingy91-fleet/releases/0.0.3%2Bd8aaaa

5. Confirm FOTA completes successfully by watching console:

[00:00:26.520,416] <inf> mflt: FOTA Update Available. Starting Download!
[00:00:26.521,820] <inf> download_client: Downloading: https://ota-cdn.memfault.com/3993/6539/11764076539?token=rFQ0Owi7Cbbf7VP9AOQimNndKD16VO_DdjLDl9ANrYg&expires=1701064800 [0]
[00:00:26.521,881] <inf> mflt: FOTA In Progress
[00:00:26.714,202] <inf> download_client: Setting up TLS credentials, sec tag count 3
[00:00:26.714,324] <inf> download_client: Connecting to https://ota-cdn.memfault.com/3993/6539/11764076539?token=rFQ0Owi7Cbbf7VP9AOQimNndKD16VO_DdjLDl9ANrYg&expires=1701064800
[00:00:28.714,447] <inf> download_client: Downloaded 1024/424428 bytes (0%)
[...]
[00:07:33.470,214] <inf> download_client: Downloaded 316416/424428 bytes (74%)
[00:07:35.357,604] <inf> app_event_manager: LOCATION_MODULE_EVT_CLOUD_LOCATION_DATA_READY
[00:07:35.358,337] <inf> app_event_manager: DATA_EVT_DATA_READY
[00:07:35.364,715] <inf> app_event_manager: DATA_EVT_CLOUD_LOCATION_DATA_SEND
[00:07:35.382,873] <inf> app_event_manager: DATA_EVT_DATA_SEND_BATCH
[00:07:35.383,758] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
[00:07:35.384,094] <inf> app_event_manager: CLOUD_EVT_CLOUD_LOCATION_UNKNOWN
[00:07:35.384,857] <inf> app_event_manager: LOCATION_MODULE_EVT_INACTIVE
[00:07:35.386,108] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
[00:07:40.273,132] <inf> download_client: Downloaded 317440/424428 bytes (74%)
[00:07:41.878,082] <err> download_client: Unexpected HTTP response: 400 bad request
[00:07:41.878,112] <err> fota_download: Download client error
[00:07:41.878,143] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
[00:07:41.878,173] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
[00:07:41.879,608] <err> mflt: FOTA failed -- trying again ...
[00:07:50.251,220] <inf> mflt: FOTA Update Available. Starting Download!
[...]
[00:10:32.878,387] <inf> download_client: Downloaded 421888/424428 bytes (99%)
[00:10:33.554,199] <inf> download_client: Downloaded 422912/424428 bytes (99%)
uart:~$ *** Booting nRF Connect SDK dcaa0fbee306 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x679f0 bytes
D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
D: erasing secondary header
D: erasing secondary trailer
I: Bootloader chainload address offset: 0xc000
I: Jumping to the first image slot

---

Resolves: MFLT-12431

[github-actions]

Some checks failed. Please fix and resubmit.

checkpatch (informational only, not a failure)

/home/runner/work/memfault-asset-tracker/memfault-asset-tracker/scripts/checkpatch.pl: 1: 404:: not found

Gitlint issues

3: B6 Body message is missing

Identity/Emails issues

3c481ed: author email (Gillian Minnehan [email protected]) needs to match one of the signed-off-by entries.

Tip: The bot edits this comment instead of posting a new one, so you can check the comment's history to see earlier messages.

[chrisc11]

@noahp I am unlikely to get to this during the week, could you commandeer to help get it through?

[noahp]

my kingdom for a single kconfig flag 🥹

[gminn]

This worked like a charm for me!! 😍 So excited. Thank you @chrisc11 for finding the solution on this one. I applied a few clean up changes but otherwise looks good to me. @memfault/owners-mcu eyes would be great! Once I merge it, I will update our quickstart guide, set up a new release in the app, and prompt the rest of the team to flash with a new image once we merge. Can't wait!

[chrisc11]

Appears I can't approve my own PR 😆 but this LGTM once CI passes. Thanks for the patch ups

[chrisc11]

nit: Do we expect some of the CI jobs to fail because we have forked the repo? If so, maybe just disable them? (cc @gminn @noahp)