[sw] polling in ROM

[tjaychen]

Certain setup routines (for example sram / flash scramble key request) require the ROM to make a hardware request and poll for its completion. Polling in ROM however, can be bad if the hardware underneath failed and the ROM just never gets out of the loop.

As a result, we should think a bit about what kind of polling-timeout scheme we want in ROM and how that should terminate in the event of a failure.

[tjaychen]

@alphan
@moidx
is this already handled? I figure you guys probably have something planned for it already.

[alphan]

Thanks @tjaychen. Currently we are not using an explicit polling-timeout scheme. Instead, we are relying on the watchdog which is initialized early in the boot with a safe default (TBD) here and reconfigured based on the life cycle state here. Also, ROM does not pet the watchdog, we plan to set the timeout to a value that is long enough for proper ROM execution.

The main reason for this approach is simplicity, i.e. no iteration counts to tune based on the clock frequency or peripheral being polled e.g. sram_ctrl, flash_ctrl, otbn, keymgr, hmac, uart, etc.

[tjaychen]

@alphan just to clarify, wouldn't a watchdog in this case actually reboot the device?
If that were to happen, will you guys get any information on what hung where? We may be able to make use of the rstmgr crash dump here as well, but i'm not sure if that information is exposed.

[alphan]

Good point, I think I was assuming debugging would provide that information.

[tjaychen]

yeah we have a rstmgr crashdump that basically is intended to show where the processor was prior to reset.
However we will have to expose that info somehow though, because if we watchdog during production state, and it's a ROM timeout, we may never get to a debug interface or a modifiable piece of software to dump out where things to stuck.

The "timeout" failure I was referring to was a failure, and more a "fall-through" of a polling loop, that would then manifest some kind of user visible error so we can figure out where to look.

All of these are possible options though.

[alphan]

I think we can close this since we now can

• Attach a debugger,
• Build programs to be executed from SRAM (e.g. for dumps over UART) and inject them.

@tjaychen @dmcardle plmk if you have any concerns.

[tjaychen]

it's not entirely the same, because the assumption is that some kind of lock-up is happening after we enter life cycle PROD, which means the debugger would no longer be available. We have seen this from time to time just because during production there's more volume and it can expose corner cases not previously thought of.

We don't necessarily need a fall through like what i'm suggesting, but just have some kind of plan of how we should debug if things were to get stuck.

[alphan]

Yeah, prod can be tricky. I think this deserves an offline discussion.

[msfschaffner]

@timothytrippel @moidx @cfrantz @cdgori should we pick this discussion up again for PROD?

[cfrantz]

As of 0539d58 (and to my best knowledge)

Known polling loops without timeouts:

• uart functions: spin waiting if TX FIFO is full.
• Some functions (none currently used in ROM) spin waiting for TX FIFO empty.

Known polling loops with timeouts:

• pwrmgr_cdc_sync spins with a CPU mcycle timeout (the timeout is the amount of time a CDC sync should take, plus a small amount of error overhead).
• shutdown's UART functions spin with a CPU mcycle timeout (the timeout is the amount of time it should take to empty the TX fifo).