Potential race condition between delegation record updates and withdrawal acknowledgements

[joe-bowman]

Summary of Bug

At the epoch boundary, the following operations happen:

• Process unbondings
• Query DelegatorDelegations (in order to ensure parity between QS records and host chain)
• Withdraw rewards

Recently there have been changes in the redemption rate outside of expected bounds; we believe this may relate to a race condition between updating delegation records and receiving unbonding tx acknowledgements.

The delegatorDelegations query is not height bound - it uses the height of 0. We do collect the height of the epoch on other chains for a boundary but this is stored outside of the interchainstaking module. This means the delegation records are collected from the current height at the point the ICQ query is made.

This combined with the ICA unbonding txs that are triggered when the unbondings are processed, it is undeterminable whether the delegation record we get is before, or after the unbonding tx hits the host chain, nor whether the response posted to the QS chain hits before or after the unbonding acknowledgement is updated.

Expected Behaviour

The expected behaviour is that we have an up to date version of the host chain delegation state reflected on QS at the point we calculate the redemption rate.

In order to achieve this, we will need to process unbondings and wait for the acknowledgements before the RR update can take place - or ensure unbondings are triggered after delegations have been updated.

[joe-bowman]

An ideal way to fix this, would be to have a 'height' field on the delegation record; as we have the height for an ICQ request, but an ICA acknowledgement does not give us a height, and therefore we cannot use it as a comparison.