Comprehensive testing of failure scenarios

[LuqiPan]

What is the Problem Being Solved?

Description of the Design

RunUtils test covering happy path and error paths (from product)

Error paths from implementation and upstream service irregularities

Race conditions

Verify vstorage output

Security Considerations

Scaling Considerations

Test Plan

Upgrade Considerations

[0xpatrickdev]

A lot of work was recently done around potential failure scenarios wrt to StatusManager. Namely, accounting for failed IBC transfers and unexpected timings around advance and settlement.

The bulk of the above was captured in #10530 which also included diagrams of the state machine:
https://github.com/Agoric/agoric-sdk/blob/23120a923c2bd3ca182291df2080214fbd5f20ca/packages/fast-usdc/README.md#status-manager

At the time of writing, there is a mistake in the Minted -> Forwarded step. Mints should wait for Observe before proceeding.

[0xpatrickdev]

At the time of writing, there is a mistake in the Minted -> Forwarded step. Mints should wait for Observe before proceeding.

This might suggest we need an additional state - perhaps Minted - to capture "minted before observed". If it's not a requirement of the state machine internals, we might want it for vstroage reporting (under a new key since we won't have the evm txHash).

This surfaced when discussing #10508 with @samsiegart. Our current plan is only to report TxStatus statuses (below) to vstorage and log the "minted before observed" events to the swingset console.

agoric-sdk/packages/fast-usdc/src/constants.js

Lines 6 to 22 in 23120a9

	export const TxStatus = /** @type {const} */ ({
	/** tx was observed but not advanced */
	Observed: 'OBSERVED',
	/** IBC transfer is initiated */
	Advancing: 'ADVANCING',
	/** IBC transfer is complete */
	Advanced: 'ADVANCED',
	/** IBC transfer failed (timed out) */
	AdvanceFailed: 'ADVANCE_FAILED',
	/** settlement for matching advance received and funds disbursed */
	Disbursed: 'DISBURSED',
	/** fallback: do not collect fees */
	Forwarded: 'FORWARDED',
	/** failed to forward to EUD */
	ForwardFailed: 'FORWARD_FAILED',
	});
	harden(TxStatus);

[dckc]

In discussion of #10572 / #10388, we agreed to postpone this one to here:

• [todo] C12 - Contract MUST only pay back the Pool only if they started the advance before USDC is minted

[0xpatrickdev]

Task List

I suggest we create separate issues for each and close this once were in agreement:

Priority	Task	Details
1	Wrap vow handlers that create promises	• Some onFulfilled and onRejected handlers can throw; these failures will be silent (no unhandled promise rejection logs) until #10576 • @dckc suggests wrapping anything that can throw in try/catch for more graceful capture
2	Boot tests for Tx status updates and Pool Metrics	• Covers requirements around visibility for happy and failure paths from off-chain perspective • All paths/states should be tested in bootstrap
3	Debug logging coverage	• vstorage writes do not generate a console log • Need observability through TX lifecycle in console logs • Current coverage is decent but could improve in: eventFeed, statusManager
4	localTransfer, withdrawToSeat failure paths	• Unexpected case but low effort to return payment to LP • Task involves wiring up this return mechanism
5	FORWARD_FAILED state	• Transition to Forwarded on success, not initation • Add state for ForwardFailed

Google Doc where much of this was fleshed out. There are many other paths around LP share path, fee calcs, and unexpected sequences we already have sufficient coverage for.

Tickets:

1. Boot Tests + Debug Logging (combined 2 +3)
2. localTransfer, withdrawToSeat failure paths
3. FORWARD_FAILED state

[0xpatrickdev]

Resulting tickets:

• fusdc: bootstrap tx observability coverage #10623
• fusdc: localTransfer, withdrawToSeat failure paths #10624
• fusdc: ForwardFailed state #10625