Reduce overestimation of minimum UTxO values for Shelley-era addresses.

[jonathanknowles]

Issue Number

ADP-2039

Summary

This PR reduces our overestimation of minimum UTxO values for outputs with Shelley-era addresses.

Details

• When computing minimum ada quantities for user-specified outputs, coin selection now uses the actual addresses specified by the user.
• When computing minimum ada quantities for auto-generated change outputs, coin selection now uses a maximum length change address that is specific to the current wallet key type.
  • This allows us to use a smaller overestimation in the case of Shelley-era addresses, as the longest Shelley-era address is shorter than the longest Byron-era address.

[jonathanknowles]

bors try

[iohk-bors]

try

---

ℹ️ Note that:

• re-running the failed jobs caused the test suite to pass.
• links to suspected flaky tests have been added below.

---

Build that originally failed:

• ci/hydra-build:required

Build constituent:

• linux.musl.checks.cardano-wallet.integration-prev-era

Log:

• https://hydra.iohk.io/build/17347893/nixlog/16

Log extract:

Failures:

  src/Test/Integration/Scenario/API/Shelley/TransactionsNew.hs:2481:26: 
  1) API Specifications, NEW_SHELLEY_TRANSACTIONS, TRANS_NEW_JOIN_01a - Can join stakepool, rejoin another and quit
       While verifying value:
         ( Status
             { statusCode = 200
             , statusMessage = "OK"
             }
         , Right
             ( ApiWallet
                 { id = ApiT
                     ( WalletId { getWalletId = b7bb92c58a5fcedae4a9312ea7e763739e07cc71 } )
                 , addressPoolGap = ApiT
                     ( AddressPoolGap
                         { getAddressPoolGap = 20 }
                     )
                 , balance = ApiWalletBalance
                     { available = Quantity 8708476
                     , total = Quantity 10566734
                     , reward = Quantity 1858258
                     }
                 , assets = ApiWalletAssetsBalance
                     { available = ApiT
                         ( TokenMap
                             ( fromList [] )
                         )
                     , total = ApiT
                         ( TokenMap
                             ( fromList [] )
                         )
                     }
                 , delegation = ApiWalletDelegation
                     { active = ApiWalletDelegationNext
                         { status = Delegating
                         , target = Just
                             ( ApiT
                                 ( PoolId
                                     { getPoolId = "»\x11L³}uú\x5&\x3(Â5£Úâ\x95£=\xb¦t¥ë\x1e>V\x8e" }
                                 )
                             )
                         , changesAt = Nothing
                         }
                     , next = []
                     }
                 , name = ApiT
                     ( WalletName
                         { getWalletName = "Empty Wallet" }
                     )
                 , passphrase = Just
                     ( ApiWalletPassphraseInfo 2022-08-02 04:15:13.27911613 UTC )
                 , state = ApiT Ready
                 , tip = ApiBlockReference
                     { absoluteSlotNumber = ApiT
                         ( SlotNo 4291 )
                     , slotId = ApiSlotId
                         { epochNumber = ApiT
                             ( EpochNo
                                 { unEpochNo = 53 }
                             )
                         , slotNumber = ApiT
                             ( SlotInEpoch
                                 { unSlotInEpoch = 51 }
                             )
                         }
                     , time = 2022-08-02 04:18:21.2 UTC
                     , block = ApiBlockInfo
                         ( Quantity 2115 )
                     }
                 }
             )
         )
       
       Waited longer than 90s to resolve action: "Wallet has consumed rewards".
       expected: Quantity 0
        but got: Quantity 1858258

  To rerun use: --match "/API Specifications/NEW_SHELLEY_TRANSACTIONS/TRANS_NEW_JOIN_01a - Can join stakepool, rejoin another and quit/"

  src/Test/Integration/Scenario/API/Shelley/StakePools.hs:1228:25: 
  2) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools
       While verifying value:
         ( Status
             { statusCode = 200
             , statusMessage = "OK"
             }
         , Right
             [ ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "ì(ó=ËæÖ@
                         \x1e^3\x9bÐd|\x9sÊl\xcù»æ\x83\x8dÆ" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 56262025621969
                     , relativeStake = Quantity
                         ( Percentage
                             ( 1163 % 10000 )
                         )
                     , saturation = 0.3489045382387637
                     , producedBlocks = Quantity 826
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPA" }
                             , name = "Genesis Pool A"
                             , description = Nothing
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 200000000000000
                 , retirement = Nothing
                 , flags = []
                 }
             , ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "»\x11L³}uú\x5&\x3(Â5£Úâ\x95£=\xb¦t¥ë\x1e>V\x8e" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 56260265394038
                     , relativeStake = Quantity
                         ( Percentage
                             ( 233 % 2000 )
                         )
                     , saturation = 0.34956332295773646
                     , producedBlocks = Quantity 802
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPD" }
                             , name = "Genesis Pool D"
                             , description = Just "Lorem Ipsum Dolor Sit Amet."
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 100000000000000
                 , retirement = Just
                     ( ApiEpochInfo
                         { epochNumber = ApiT
                             ( EpochNo
                                 { unEpochNo = 1000000 }
                             )
                         , epochStartTime = 2023-02-03 08:30:43 UTC
                         }
                     )
                 , flags = []
                 }
             , ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "´WhÁ¢ÚKÑ>¼ª\x1e¥\x14\x8í£\x1dÌ!v\ËÔ\x7Í©ò" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 55677257462493
                     , relativeStake = Quantity
                         ( Percentage
                             ( 993 % 10000 )
                         )
                     , saturation = 0.29779522143257986
                     , producedBlocks = Quantity 691
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPC" }
                             , name = "Genesis Pool C"
                             , description = Just "Lorem Ipsum Dolor Sit Amet."
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 100000000000000
                 , retirement = Just
                     ( ApiEpochInfo
                         { epochNumber = ApiT
                             ( EpochNo
                                 { unEpochNo = 100000 }
                             )
                         , epochStartTime = 2022-08-20 16:30:43 UTC
                         }
                     )
                 , flags = []
                 }
             ]
         )
       
       Waited longer than 300s to resolve action: "metadata is fetched".
       expected: [PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}]
        but got: []

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools/"

Randomized with seed 674796091

Related issues:

• TRANS_NEW_JOIN_01a
  Flaky Test: STAKE_POOLS_JOIN_01rewards, Can join a pool, earn rewards and collect them #2415
• STAKE_POOLS_SMASH_01
  Flaky STAKE_POOLS_SMASH_01 - Waited longer than 90s to resolve action: "metadata is fetched" #2331

[Anviking]

With

txOutputMinimumAdaQuantity txConstraints
                 (AD.maxLengthAddressFor $ Proxy @ShelleyKey)
                 TokenMap.empty

this seems either incorrect or misleading. It's not the minimum value: minLengthAddress would yield a smaller result. I think we should either

1. Replace AD.maxLengthAddressFor with minLengthAddress
2. Reword this paragraph.

(And arguably the 4*44 lovelace overestimation of the requirement for coin values (below 4295 ada) is also at odds with this value being an "absolute minimum", though only when considering the broader context outside of the wallet API.)

[jonathanknowles]

Thanks @Anviking, I think you've raised a good point.

I'll have a think about what we should do here, and make a PR to either adjust the comment, or adjust the definition in the way that you suggest.

I think the intention of the comment is that "If you're sending ada (and no other assets) to a Shelley address, then sending this amount of ada or more should always work when creating transactions with the wallet". So in this sense, it's the minimum value that we can safely state will always be fine for all Shelley addresses, when using the wallet.

If we were to alter the definition to use minLengthAddress, but then the user specifies a Shelley address that is longer, then their payment would be rejected (if they had chosen to pay the value returned by this field). I would argue that this would make the min_utxo_value field even less useful that it already is.

As you know, I think we should mark this field as "deprecated" as soon as possible, since it's quite hard for the user to interpret the meaning (and of course, this protocol parameter no longer exists in the Alonzo and Babbage eras).

[jonathanknowles]

@Anviking I've added a reminder to address this comment here: https://input-output.atlassian.net/browse/ADP-2039.

[Anviking]

If you're sending ada (and no other assets) to a Shelley address, then sending this amount of ada or more should always work when creating transactions with the wallet

I think that's fair, but this is what should be said in the docs then. So e.g. something more like:

In Babbage the minimum, even for ada-only outputs, is not constant. It depends upon the serialised length of the output, including address and ada quantity. What is returned by this field is a value which works for sending ada to any Shelley address. The true minimum may be marginally smaller with short addresses, or bigger with longer Byron addresses or included tokens. The field may be deprecated or removed in future versions of the wallet.

I think we should mark this field as "deprecated" as soon as possible, since it's quite hard for the user to interpret the meaning

I do agree 💯

[Anviking]

of the in-memory representation of that output, which was not dependent on the length of the address.

Are you sure this is entirely correct? I thought it was an upper bound of the in-memory representation.

[jonathanknowles]

of the in-memory representation of that output, which was not dependent on the length of the address.

Are you sure this is entirely correct?

This was my understanding, after previous discussions with the ledger team.

I thought it was an upper bound of the in-memory representation.

This would indeed explain why the address length does not affect the Alonzo-era minimum UTxO value computation.

I'll double-check with the ledger team and/or verify this statement by looking at the source code.

If it turns out that this description is inaccurate, I'll make another PR to fix it. 👍🏻

[jonathanknowles]

@Anviking I've added a reminder to address this comment here: https://input-output.atlassian.net/browse/ADP-2039.

[Anviking]

lgtm 🎉 modulo in particular #3416 (comment)

[jonathanknowles]

bors r+

[iohk-bors]

Build failed:

• ci/hydra-build:required

Failures:

  src/Test/Integration/Scenario/API/Shelley/StakePools.hs:1185:21: 
  1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_GARBAGE_COLLECTION_01 - retired pools are garbage collected on schedule and not before
       predicate failed on: 6
       Waited longer than 90s to resolve action: "Garbage has been collected for epoch 7.".

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_GARBAGE_COLLECTION_01 - retired pools are garbage collected on schedule and not before/"

Randomized with seed 763525993

#2337

[jonathanknowles]

bors r+

[iohk-bors]

Build succeeded:

• buildkite/cardano-wallet
• ci/hydra-build:required