pendingchannels rpc gives wrong htlc stage and sweeps in 2 separate transactions after turning on lnd

[antonilol]

Background

Describe your issue here.

i made a channel 800k sats and keysended 50k sats to the other side. there were no pending HTLCs at this point in time and remote balance was 50k sats. then, i made 2 HTLCs of 50k sats each (with a swap), turned off my node and gave the remote party the preimage of 1 of the 2 swaps (by redeeming on-chain). the other party will try to fulfill the HTLC but i am not online, so force closed the channel before it times out to claim it on chain.

when i turned the node back on, it rescanned the chain and obviously saw the force close, it directly swept the local output (:3), it took some time before it swept the timed out HTLC (:2) and in between a block was mined (dont know it this matters though)

before the timed out HTLC was swept the pendingchannels rpc said it was on stage 2 (after the sweep confirmed it wasnt pending anymore)

{
    "total_limbo_balance": "749730",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
        {
            "channel": {
                "remote_node_pub": "03f060953bef5b777dc77e44afa3859d022fc1a77c55138deb232ad7255e869c00",
                "channel_point": "23e8ac05ab5e50977e65224efaf66a228bd2115814afafb86bf22c84d526e022:1",
                "capacity": "800000",
                "local_balance": "649730",
                "remote_balance": "0",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "STATIC_REMOTE_KEY",
                "num_forwarding_packages": "0",
                "chan_status_flags": ""
            },
            "closing_txid": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f",
            "limbo_balance": "749730",
            "maturity_height": 2196616,
            "blocks_til_maturity": -46,
            "recovered_balance": "0",
            "pending_htlcs": [
                {
                    "incoming": false,
                    "amount": "50000",
                    "outpoint": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f:1",
                    "maturity_height": 2196625,
                    "blocks_til_maturity": -37,
                    "stage": 2
                },
                {
                    "incoming": false,
                    "amount": "50000",
                    "outpoint": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f:2",
                    "maturity_height": 2196625,
                    "blocks_til_maturity": -37,
                    "stage": 2      <-- this
                }
            ],
            "anchor": "LIMBO"
        }
    ],
    "waiting_close_channels": [
    ]
}

after a block, the fulfilled HTLC (:1) had stage 0 for some reason, nothing happened to that output in this block, also the local sweep got mined, the timed out HTLC sweep wasnt made yet, note the recovered balance == the local balance

{
    "total_limbo_balance": "50000",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
        {
            "channel": {
                "remote_node_pub": "03f060953bef5b777dc77e44afa3859d022fc1a77c55138deb232ad7255e869c00",
                "channel_point": "23e8ac05ab5e50977e65224efaf66a228bd2115814afafb86bf22c84d526e022:1",
                "capacity": "800000",
                "local_balance": "649730",
                "remote_balance": "0",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "STATIC_REMOTE_KEY",
                "num_forwarding_packages": "0",
                "chan_status_flags": ""
            },
            "closing_txid": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f",
            "limbo_balance": "50000",
            "maturity_height": 2196616,
            "blocks_til_maturity": -47,
            "recovered_balance": "699730",
            "pending_htlcs": [
                {
                    "incoming": false,
                    "amount": "50000",
                    "outpoint": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f:1",
                    "maturity_height": 2196616,
                    "blocks_til_maturity": -47,
                    "stage": 0
                },
                {
                    "incoming": false,
                    "amount": "50000",
                    "outpoint": "de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f:2",
                    "maturity_height": 2196625,
                    "blocks_til_maturity": -38,
                    "stage": 2
                }
            ],
            "anchor": "LIMBO"
        }
    ],
    "waiting_close_channels": [
    ]
}

another block was mined with the timed out HTLC sweep, and the channel was fully closed on my side

Your environment

• version of lnd lnd version 0.14.2-beta commit=v0.14.2-beta
• which operating system (uname -a on *Nix) Linux node 5.16.14-arch1-1 #1 SMP PREEMPT Fri, 11 Mar 2022 17:40:36 +0000 x86_64 GNU/Linux
• version of btcd, bitcoind, or other backend Bitcoin Core version v22.0.0
• any other relevant environment details
  on testnet

Steps to reproduce

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

what i did in 'Background', for the swap i used https://testnet.boltz.exchange/

Expected behaviour

Tell us what should happen

• correct stage and blocks_til_maturity
• one sweep tx

Actual behaviour

Tell us what happens instead

• the strange this with the stage and blocks_til_maturity of de55d4e3b8c8d541b85cc51666f19b5acedb8a31dd308e9bc6b3ee1e899e028f:1
• the local p2wpkh output was swept separately from the timed out HTLC, which were both spendable when i turned on lnd

[tvolk131]

Also, what's going on with blocks_til_maturity? The first HTLC changed from -37 to -47.

[antonilol]

Also, what's going on with blocks_til_maturity? The first HTLC changed from -37 to -47.

i dont know, it was already spent on chain when i booted up my lnd. maybe it was trying to time it out and found out later it has been spent by the remote party

[tvolk131]

Weird... I should have some time to dig into this later tonight or tomorrow.

[Crypt-iQ]

This is because the htlc's are sent to the utxonursery which reports that everything is in stage 2. The change in stage is because the utxonursery requires a block confirmation to clean-up its entries (seen in graduateClass). I don't think this will get addressed since it is:

• cosmetic
• the utxonursery may get deleted (see utxonursery: delete #3688)

also, static-remote-key channels are on the way towards deprecation