Multiple peers unilaterally closing immediately upon new channel confirmation: "rejected commitment"

[whitslack]

Issue and Steps to Reproduce

I opened twelve channels in quick succession, and five of them were unilaterally closed by the peer as soon as they reached 3 confirmations. The debug logs for all five contain a message that says rejected commitment with an invalid_commit_sig. Also I see peer_in WIRE_ERROR, which to me says that the peer didn't like something my node did.

Here are the debug logs from four of the five failed channels. (The fifth log contains many megabytes of channel update messages.) I'm not sure why there are so many SKIPPED log events scattered throughout them, but maybe there is enough intact information across all of them to astertain what went wrong.

• 03d480ac7ad220371f1b9c2560be7eb8da6335d6a0d8cb1bc27db1a3958fd8c73f.log
• 029edbb304c7a00fbe9c912eebb8a865afe96a19015e0548e8be1691f965f1ca97.log
• 0351876efe01167a1f81f988ef470311eca17661dd1994bad01861d4a54565bc5c.log
• 03271338633d2d37b285dae4df40b413d8c6c791fbee7797bc5dc70812196d7d5c.log

For the sake of making this searchable, here's a summary of the sequence of events I see in these logs. These might be slightly out of order because I had to piece them together from multiple logs due to all the SKIPPED events.

1. Funding tx … depth 3 of 3
2. State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_NORMAL
3. peer_out WIRE_FUNDING_LOCKED
4. Setting REMOTE feerate to 17890
5. Feerate: SENT_ADD_HTLC->SENT_ADD_COMMIT REMOTE now 17890
6. peer_out WIRE_UPDATE_FEE
7. peer_out WIRE_COMMITMENT_SIGNED
8. peer_in WIRE_ERROR
9. Peer transient failure in CHANNELD_NORMAL: channeld: (ignoring) channel …: rejected commitment: commit_height=1, invalid_commit_sig=…, commit_tx=…, sig_hash=…

getinfo output

$ lightning-cli getinfo | jq '{binding, version, blockheight, network, "lightning-dir"}'
{
  "binding": [
    {
      "type": "ipv6",
      "address": "::",
      "port": 9735
    },
    {
      "type": "ipv4",
      "address": "0.0.0.0",
      "port": 9735
    }
  ],
  "version": "0.8.1",
  "blockheight": 622039,
  "network": "bitcoin",
  "lightning-dir": "/var/lib/lightning/bitcoin"
}

[whitslack]

Ah ha! Just got three more, and this time I managed to capture the logs immediately so there are no SKIPPED entries.

• 03ec421539cf213e8df9080515e6993f2a6aecec7c4db16d34514ce2ed7724cb18.log
• 023ec3d1fa35f7fb8996374cf1848c1a40788df013551c5510c75617222bd2dd2d.log
• 03271562ad6fbb9b208eafe203216233cd756a3981c77ef1c5afda109df8be5ac2.log

[niftynei]

In all 3 of the logs you sent, the peer closed the channel on you. Seems like something about the feerate update that we send on channel open isn't being well understood by the peer, as they're rejecting the subsequent signature on the commitment update.

The basic flow is:

• The channel opens "log": "peer_out WIRE_FUNDING_LOCKED"
• clightning decides to update the feerate "log": "Setting REMOTE feerate to 5034"
• clightning creates and sends a new commitment signature (using the new feerate)
• the peer rejects the updated commitment signature "log": "Peer transient failure in CHANNELD_NORMAL: channeld: (ignoring) channel cf27ceadc6d3b6a1473f3da92f89301d4465e0416ab29a0e75b8c833a2d7709f: rejected commitment: commit_height=1, invalid_commit_sig=304402202f6a58cde11173b5df32d5c1e913d75a3a21c05ba1a2cbd1bfa16f016c2d3379022075a59d54b3e5df28dec46aaf91aef9f418062ccf7e7f9f949cdbed449f44c8b8, commit_tx=0200000001cf27ceadc6d3b6a1473f3da92f89301d4465e0416ab29a0e75b8c833a2d7709f000000000048819480011e290f0000000000160014022d2657c1b2080c29babffc5f628af1401e60ca77872020, sig_hash=6cabe69f87b30ed427568772eaeaec6860afc9df01ac56699355c61177b5cd31"
• we mark it as a temporary failure, and try again a few seconds later "log": "Will try reconnect in 60 seconds"
• we reconnect, they close the channel "log": "Resolved FUNDING_TRANSACTION/FUNDING_OUTPUT by THEIR_UNILATERAL (31ebb7c854b0608455595e098adb356dcf338906c229f0718ae1b810e223a439)"

... for 8 of your channels. Seems like there's a feerate bug interop problem here. Thanks for reporting.

[niftynei]

Digging a bit more, the commitment tx that your peer is expecting a signature for doesn't have the feerate update applied. (The original commitment tx amount is still shown in the commitment tx sent back in the error message from your peer).

This seems very possibly related to a (recently) patched known bug in LND .

[whitslack]

@niftynei: Thank you for that analysis. I had a hunch it was due to an LND bug. 😉