Clear pending transactions API command

[ch1bo]

Why

Processing transactions in a Hydra head requires each node to agree on transactions. The protocol will validate transactions (on NewTx command) against it's local view of the ledger state, using the passed --ledger-protocol-parameters. As transactions can be valid or invalid based on configuration (or to some extent exact build versions of hydra-node), it is possible that one node accepts a transaction, while the peer nodes do not.

Currently, this means that the node which accepted the transaction now has a different local state than the other nodes and might try to spend outputs that other nodes don't see available. For example, when using hydraw, the node would be using outputs introduced by the previous pixel paint transaction, but other nodes will deem any new transaction invalid with a BadInputs error.

Within this feature, we want to improve the UX of hydra-node in presence of such misalignments.

Anotther scenario is that when a subset of nodes see SnapshotConfirmed but others do not. In this case we may also need an abillity to ask all the nodes to go back to the most recently agree snapshot, and continue from there. (Question: Do we have the old snapshots to go back to? If not, can we get them?) (Edit: We don't keep old snapshots, so maybe this isn't possible.)

What

• Add an API query to the hydra-node which includes pending transactions (from localTxs in the head state)
• Add a API command to the hydra-node which allows to clear pending transactions
  • This basically means to reset the localUTxO to the last confirmed snapshot utxo and empty localTxs
  • This also means that a new snapshot is requested if you're the snapshot leader
• Update documentation to explain this situation and when to use this command
• Confirm that this fixes Investigate resiliance of hydra when one node goes offline and a tx is submitted #1792

TBD

• Make the hydra-node not persist the local view instead and achieve the same by restarting nodes? Not easily doable our present event stream setup.

• This idea above does require a coordinated action by all operators. Use a checkpoint like transaction on chain to do this instead?

  • i.e. post the last confirmed snapshot state, which makes all nodes reset their local state upon observing it
  • Users seem fine with needing to do this deliberately ( see Heads stuck in a state without being able to progress snapshots #1773 )

• Also: Could fetch latest confirmed snapshots from other peers on startup and if they have a valid multi-signature adopt them. No; a node should not need to trust other peers.