Skip to content

Latest commit

 

History

History
132 lines (110 loc) · 44.9 KB

2023-03-27 Audit - Osmosis Token Transfer Analysis Testing.md

File metadata and controls

132 lines (110 loc) · 44.9 KB
Raw

Osmosis ibc hooks & smart contracts analysis and testing

In order to determine the quality and analyze all the aspects of created transfer logic by Osmosis team, including ibc hooks module and 2 smart contracts, the current design is measured against ics-20 properties that are expected to be met in every ics-20 transfer created. In addition, e2e test scenarios are created and executed to confirm the state of the design and its congruence with ics-20 specifications.

ICS-20 fungible token transfer properties

The desired properties of the ics-20 fungible token transfer:

  • Preservation of fungibility (two-way peg).
  • Preservation of total supply (constant or inflationary on a single source chain & module).
  • Permissionless token transfers, no need to whitelist connections, modules, or denominations.
  • Symmetric (all chains implement the same logic, no in-protocol differentiation of hubs & zones).
  • Fault containment: prevents Byzantine-inflation of tokens originating on chain A, as a result of chain B's Byzantine behaviour (though any users who sent tokens to chain B may be at risk).

Preservation of fungibility

The ics-20 transfer is performed and in addition to it the swapping of tokens using ibc hooks module and two smart contracts. The key points in design and implementation regarding property of preservation of fungibility in ics-20 transfer are the following:

The issues we discovered that could affect this ics-20 property are:

Preservation of total supply

The sum of total supply before and after the execution flow: transfer-swap-transfer is preserved. The design of this particular transfer overrides the base ics-20 transfer and is created with a few useful means to handle supply invariance:

  • In case of failed execution of any of the contracts, recovery of funds is obtained (Failed delivery handling)
  • In case of an incorrect format of packet memo field, function from core ibc is called to complete the transfer as it is without the contract (the swap) (e.g. in OnRecvPacketOverride())
  • Core calculation of token swapping is performed in such a way to keep the total supply unchanged (Swap calculation)

The issues discovered in this audit that could have an impact on total supply preservation:

Permissionless token transfers, no need to whitelist connections, modules, or denominations.

Since Osmosis team did not change the permission policy regarding creating channels or any other aspect of ibc relaying, we consider this ics-20 property to be guaranteed. The only part of the audited code that could be considered connected to this property is the denomination creation on Osmosis chain (Denomination on receiver chain), which has been done in accordance to ics-20 specs, particularly the denom representation.

Symmetric (all chains implement the same logic, no in-protocol differentiation of hubs & zones).

The ics-20 packet memo field is the key feature being used in the created Osmosis solution to swapping tokens. All the chains requesting the swap on Osmosis chain have to follow certain rules for creating memo field (json format, the exact key, etc.). If the memo field is included, the Osmosis chain does the appropriate validation of its format and data written in it. Other aspects of fungible token transfer changed on Osmosis chain include override of packet handling methods from ibc, but do not change the core ibc protocol (core ibc methods also called from the overridden ones). We consider these two points of the current design the crucial ones to guarantee the symmetry between the chains in terms of ibc fungible token transfer.

The issues discovered during this audit that could break this symmetry:

Fault containment

Prevents Byzantine-inflation of tokens originating on chain A, as a result of chain B's Byzantine behaviour (though any users who sent tokens to chain B may be at risk). In other words, Any byzantine chain B can’t influence the token circulation on chain A in any way, except the tokens that have been sent explicitly via token transfer from A to B. Wrt. those tokens anything can happen, and they may be lost under a byzantine behavior of B. The flow that occurs on Osmosis chain, which in this case acts as the receiving chain, can be observed in two phases: the ibc methods override and the execution of smart contracts. The methods from ibc that are overridden by Osmosis (OnRecvPacket(), SendPacket(), OnTimeoutPacket(), OnAcknowledgmentPacket()) mainly consist of validations of the received packet and its data and then sending the messages to smart contracts. No observable influence on chain A token circulation is confirmed, nor we found any during code inspection, integration, and e2e testing. The contract execution also has 2 steps: the crosschain swaps contract receives the message and forwards it to the swaprouter contract, whereas swaprouter contract does the actual swap. Since all these actions are contained on chain B (Osmosis chain in this case) we do not find them eligible to break the guarantee of Fault containment property.

Another point to carefully inspect was the fact that after the smart contracts are executed, a new ibc transfer is created and sent to chain A. Now, Osmosis chain acts as the sender chain, so the fault containment property should be guaranteed on chain A as well, in regards to the complete flow of send-swap-receive. However, chain A in this case is only responsible for keeping the format of the sending packet memo field and returning the appropriate response upon receiving the transfer from Osmosis chain. There is no particular processing of the received transfer on chain A, and it is also considered that the property of Fault containment is preserved.

End-to-end testing

To complete this phase in the most valuable way for the team, after consulting with some of the experts for ibc, fungible token transfer, and testing environments on local machines, the testing environment provided by Osmosis team was a bit tweaked, and a few additional steps added in order to execute the prepared test cases.

Environment setup

The setup for Osmosis swap provided could be improved with a few small changes:

  • Defining the VALIDATOR address to avoid receiving errors for "non-existing key":
VALIDATOR=$(osmosisd keys show validator -a --keyring-backend test)
  • The command to create pool on chain B (some errors also showed when using the provided command):
chainB tx gamm create-pool --pool-file sample_pool.json --from validator --yes  -b block --keyring-backend test --gas auto --gas-prices 0.1uosmo --gas-adjustment 1.3
  • Modify the message to instantiate crosschain contract:
MSG=$(jq --null-input \
    --arg gov "$VALIDATOR" \
    --arg sc "$SWAPROUTER_ADDRESS" \
    --arg cid "$CHANNEL_ID" \
    '{"governor":$gov, "swap_contract": $sc, "channels": [["osmo", $cid]]}')

Hermes setup

  1. In order to be able to execute the relaying process manually, step by step (relay, receive, ack) we removed the automatic start of the relayer in the script osmosis/tests/localrelayer/scripts/setup_hermes.sh
echo "✉️ Start Hermes"
hermes start
  1. We added a sleep infinity, just to start the container, and leave the relaying actions for later manual execution:
echo "🔌 Keeping the container on"
echo "📬 Operate Hermes manually"
sleep infinity
  1. We used the Hermes CLI commands to send relaying transactions/queries. The explanation and various commands that can be used with hermes can be found here. The most commonly used ones are the following:
hermes query packet pending --chain localosmosis-b --port transfer --channel channel-0
hermes tx packet-recv --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel channel-0
hermes tx packet-ack --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel channel-0
hermes query transfer denom-trace --chain localosmosis-b --hash ED07A3391A112B175915CD8FAF43A2DA8E4790EDE12566649D0C2F97716B8518
hermes query tx events --chain localosmosis-b --hash AA1783800C810F5B8B00E8BF6161C32691BFFABF3D248137045C64FF0446D5FD
hermes listen --chain localosmosis-a

Besides using hermes manually, we also had some test cases where the idea was to simulate the unavailable relayer, or its failure. In these cases stopped the hermes docker container, while localosmosis and faucet containers were still running. This gave us an opportunity to observe the behaviour of a chain in different conditions when relayer is not available.

In addition, several log outputs, events, if conditions, and other code changes were added to the repo after which the complete enivronment was rebuilt to inspect the desired behaviours. In order to achieve this, the complete environment needs to be built from the ground up, and for this, we recomment using the following procedure:

  • Delete both .osmosisd-local-a and .osmosisd-local-b folders created after the first build
  • Delete all osmosis and hermes docker images
  • Build the environment but instead of using make build, use:
DOCKER_BUILDKIT=1 docker-compose build --no-cache

Test cases (e2e)

Created test cases with short description and observed outputs are shown in the table below.

Description Balances
before		 Balances
after		 Result Commands
list
1. - SendTransfer
- Stop Hermes
- OnTimeoutPacket exec
- OnRecvPacket not exec		 Sender: 9899976905uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000 : 1000000
Escrow: 100000000		 Sender: 9899965837uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000 : 1000000
Escrow: 100000000		 - timeout timestamp: 10s
- gas: 11068
- sender: -11068		 $MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"200","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver": $VALIDATOR, "on_failed_delivery": "do_nothing"}}}}')
$chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 200uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO" --packet-timeout-timestamp 10000000000
#hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
2. - SendTransfer
- OnRecvPacket Timeout
- OnTimeoutPacket chain Ok
- Stop Hermes
- OnTimeoutPacket SC Fail
- Start Hermes		 Sender: 9899958354uosmo, 6686ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1006800uosmo : 993314ibc
Escrow: 100006800uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 9899947117uosmo, 6686ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1006800uosmo : 993314ibc
Escrow: 100006800uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11237uosmo
- sender: −11237uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"6900","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 6900uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO" --packet-timeout-timestamp 30000000000
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min - Timeout
~ Stop Hermes container
~ Sleep Over - Timeout
~ Start Hermes container
3. - SendTransfer
- OnRecvPacket Failed
- Stop Hermes
- OnAckPacket(failed) not exec
-Restart Hermes		 Sender: 9799887028uosmo : 294ibc
Receiver IbcHooks: -
Crosschain SC: 1000ibc
Swaprouter SC: -
Pool: 1000300ibc : 999706uosmo
Escrow: 200001400uosmo		 Sender: 9799875567uosmo : 294ibc
Receiver IbcHooks: -
Crosschain SC: 1000ibc
Swaprouter SC: -
Pool: 1000300ibc : 999706uosmo
Escrow: 200001400uosmo		 - gas: 10482
- VALIDATOR A: -11461
- VALIDATOR B: 0
- ESCROW: 0		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"100","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 100uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
~ 60s sleep before returning from OnRecvPacket
4. - SendTransfer
- OnRecvPacket Timeout
- Stop Hermes
- OnTimeoutPacket not exec
-Restart Hermes		 Sender: 9799840984uosmo : 392ibc
Receiver IbcHooks: -
Crosschain SC: 1000ibc
Swaprouter SC: -
Pool: 1000400ibc : 999608uosmo
Escrow: 200001600uosmo		 Sender: 9799829523uosmo : 392ibc
Receiver IbcHooks: -
Crosschain SC: 1000ibc
Swaprouter SC: -
Pool: 1000400ibc : 999608uosmo
Escrow: 200001700uosmo		 - gas: 11461
- VALIDATOR A: -11461
- VALIDATOR B: 0
- ESCROW: 0		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"100","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 100uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO" --packet-timeout-timestamp 30000000000
~ 60s sleep > timeout
~ Hermes stopped
5. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket router SC
Fail before saving
to SWAP_REPLY_STATES
- Stop Hermes
-Restart Hermes		 Sender: 4899976466uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899965194uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000111uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11161uosmo
- sender: -11272uosmo
- escrow +111uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +111uosmo
Escrow -111uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"111","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": "do_nothing"}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 111uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
6. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket router SC Fail
after saving to SWAP_REPLY_STATES
- Stop Hermes
-Restart Hermes		 Sender: 4899954144uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899942761uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000222uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11161uosmo
- sender: -11383uosmo
- escrow +222uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +222uosmo
Escrow -222uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"222","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": "do_nothing"}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 222uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
7. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket router
SC Fail on BankMsg:Send
- Stop Hermes		 Sender: 98 ibs/hash | 4899976171 uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000100 ibc/hash | 999902 uosmo
Escrow: 100000000		 Sender: 98 ibs/hash | 4899964703 uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000100 ibc/hash | 999902 uosmo
Escrow: 100000100		 The memo is filled with a large number for swap amount.
The transfer failed on calling contract execution.
Tokens are left on escrow address.		 MEMO=$(jq --null-input --arg cc "$CROSSCHAIN_SWAPS_ADDRESS" --arg val "$VALIDATOR" '{"wasm": {"contract": $cc, "msg": {"osmosis_swap":{"swap_amount":"1000000000000","output_denom":"uosmo","slippage":{"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$val, "on_failed_delivery": "do_nothing"}}}}')
chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 100uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
8. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain
SC Fail before saving
to SWAP_REPLY_STATES
- Stop Hermes
-Restart Hermes		 Sender: 4899942983uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899931489uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000333uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11161uosmo
- sender: -11494uosmo
- escrow +333uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +333uosmo
Escrow -333uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"333","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": "do_nothing"}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 333uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
9. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC Fail
after saving to SWAP_REPLY_STATES
(and after deleting)
- Stop Hermes
-Restart Hermes		 Sender: 4899942983uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899942983uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11161uosmo
- sender: -11494uosmo
- escrow +333uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +333uosmo
Escrow -333uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"1500","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": "do_nothing"}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 1500uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
10. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC Fail
between two replies
- Stop Hermes
-Restart Hermes		 Sender: 4899898258uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899884515uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100002500uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11243uosmo
- sender: −13743uosmo
- escrow +2500uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +2500uosmo
Escrow -2500uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"2500","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 2500uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
11. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC Fail
in handle_forward_reply
- Stop Hermes		 Sender: 4899887015uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899872272uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100003500uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11243uosmo
- sender: −14743uosmo
- escrow +3500uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +3500uosmo
Escrow -3500uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"3500","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 3500uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
12. - SendTransfer

- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC
(ibc transfer M2)
- Relay M2
- Ack M1		 Sender: 4899987625uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo		 Sender: 4899976171uosmo | 98ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 999902uosmo : 1000100ibc
Escrow: 100000100uosmo		 -gas: 11353
-sender: -11454uosmo
-escrow: +100uosmo

Sender received 98uosmo according to swap.
Escrow balance increased by 100 uosmo.		 $MEMO=$(jq --null-input --arg cc "$CROSSCHAIN_SWAPS_ADDRESS" --arg val "$VALIDATOR" '{"wasm": {"contract": $cc, "msg": {"osmosis_swap":{"swap_amount":"100","output_denom":"uosmo","slippage":{"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$val, "on_failed_delivery": "do_nothing"}}}}')
$chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 100uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
#hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel channel-0
#hermes tx packet-recv --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel channel-0
#hermes tx packet-ack --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel channel-0
#hermes tx packet-ack --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel channel-0
13. - SendTransfer
- OnRecvPacket Ok
- OnAckPacket Ok
- OnAckPacket SC Fail		 Sender: 9899987626uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 9899969591uosmo, 6686ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1006800uosmo : 993314ibc
Escrow: 100006800uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11235uosmo
- sender: −18035uosmo, +6686ibc
- escrow +6800uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"6800","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 6800uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
# hermes tx packet-ack --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel "channel-0"
14. - SendTransfer
- OnRecvPacket Ok
- OnTimeoutPacket Ok
- OnTimeoutPacket SC Fail		 Sender: 9899969591uosmo, 6686ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1006800uosmo : 993314ibc
Escrow: 100006800uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 9899958354uosmo, 6686ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1006800uosmo : 993314ibc
Escrow: 100006800uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11237uosmo
- sender: −11237uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"6800","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 6800uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO" --packet-timeout-timestamp 30000000000
# hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
15. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC
(ibc transfer M2) Ok
- …
- Stop Hermes before Ack M2
- Ack M2		 Sender: 4899976171uosmo | 98ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 999902uosmo : 1000100ibc
Escrow: 100000100uosmo		 Sender: 4899964716uosmo | 196ibc
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 999804uosmo : 1000200ibc
Escrow: 100000200uosmo		 Balances are checked after the hermes was stopped.
- gas: 11354
- sender: -11455
- escrow: +100
After restarting hermes, ack is in unreceived acks.
It can be relayed using manual commands.		 $MEMO=$(jq --null-input --arg cc "$CROSSCHAIN_SWAPS_ADDRESS" --arg val "$VALIDATOR" '{"wasm": {"contract": $cc, "msg": {"osmosis_swap":{"swap_amount":"100","output_denom":"uosmo","slippage":{"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$val, "on_failed_delivery": "do_nothing"}}}}')
$chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 100uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
#hermes tx packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel channel-0
#hermes tx packet-ack --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel channel-0
#hermes tx packet-recv --dst-chain localosmosis-a --src-chain localosmosis-b --src-port transfer --src-channel channel-0
Stop hermes
Restart hermes
#hermes tx packet-ack --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel channel-0
16. - SendTransfer
- OnRecvPacket WasmHook Ok
- OnRecvPacket Router SC Ok
- OnRecvPacket Crosschain SC Fail
in handle_forward_reply – LocalRecoveryAddr
- Restart Hermes during OnRecvPacket		 Sender: 4899875772uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100000000uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 Sender: 4899861029uosmo
Receiver IbcHooks: -
Crosschain SC: -
Swaprouter SC: -
Pool: 1000000uosmo : 1000000ibc
Escrow: 100003500uosmo
Swap Reply State SR: -
Swap Reply State CCS: -
Forward Reply State CCS: -		 - gas: 11243uosmo
- sender: −14743uosmo
- escrow +3500uosmo

After restarting Hermes, packet is in the unreceived packets. Balances:
Sender +3500uosmo
Escrow -3500uosmo		 $ MEMO=$(jenv -c '{"wasm": {"contract": $CROSSCHAIN_SWAPS_ADDRESS, "msg": {"osmosis_swap":{"swap_amount":"3500","output_denom":"uosmo","slippage": {"twap": {"slippage_percentage":"20", "window_seconds": 10}},"receiver":$VALIDATOR, "on_failed_delivery": {"local_recovery_addr":$VALIDATOR}}}}}')
$ chainA tx ibc-transfer transfer transfer $CHANNEL_ID $CROSSCHAIN_SWAPS_ADDRESS 3500uosmo --from validator -y "${TX_FLAGS[@]}" --memo "$MEMO"
# hermes tx <br>packet-recv --dst-chain localosmosis-b --src-chain localosmosis-a --src-port transfer --src-channel "channel-0"
~ Sleep 1min
~ Stop Hermes container
~ Start Hermes container
~ Sleep Over