can't close channel in state "CHANNELD_SHUTTING_DOWN" #1865

martijnbolt · 2018-08-22T19:02:48Z

Issue and Steps to Reproduce

I cannot close a channel that is in a state "CHANNELD_SHUTTING_DOWN" for ages now. For some reason it seems as if the api does not accept this unilateral close and tries to make it bilateral which the nodes can't negotiate, bringing me to a kind of "deadlock" situation. I guess an api call like "froce close" would be good since publishing my commitment tx should always work

Steps to reproduce:
https://imgur.com/a/rYUlEu6

`getinfo` output

The text was updated successfully, but these errors were encountered:

cdecker · 2018-08-23T17:03:38Z

How long have you been waiting for the closing state to resolve? In recent versions the channel should eventually fall back to a unilateral close automatically. What version are you running?

martijnbolt · 2018-08-24T08:58:51Z

v0.5.2-2016-11-21-2069-gb8c6365
aka. Bitcoin Savings & Trust Daily Interest II

it's old I guess... I'll install a new version and let you know.

martijnbolt · 2018-08-24T14:49:30Z

I now run version: v0.6-392-g898655f

This is the problematic channel now:
`

"peers": [
{
"id": "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432",
"connected": false,
"channels": [
{
"state": "AWAITING_UNILATERAL",
"short_channel_id": "506902:2127:1",
"channel_id": "fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c",
"funding_txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
"msatoshi_to_us": 20000000,
"msatoshi_to_us_min": 20000000,
"msatoshi_to_us_max": 20000000,
"msatoshi_total": 20000000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709551615,
"their_channel_reserve_satoshis": 0,
"our_channel_reserve_satoshis": 200,
"spendable_msatoshi": 19800000,
"htlc_minimum_msat": 0,
"their_to_self_delay": 144,
"our_to_self_delay": 144,
"max_accepted_htlcs": 483,
"status": [
"Loaded from database"
],
"in_payments_offered": 0,
"in_msatoshi_offered": 0,
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"out_payments_offered": 0,
"out_msatoshi_offered": 0,
"out_payments_fulfilled": 0,
"out_msatoshi_fulfilled": 0
}
]
},
{
"id": "034a5fdb2df3ce1bfd2c2aca205ce9cfeef1a5f4af21b0b5e81c453080c30d7683",
"connected": false,
"channels": [
{
"state": "CHANNELD_NORMAL",
-----------------snip---------------

`
I'm using the 'close' command to try and close the channel (I tried this with channel_id and peer_id). This hangs my lightning-cli:.

pi@retropie:~/clightning/lightning/lightningd $ sudo /home/pi/lightning/cli/./lightning-cli close fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c | jq [.]
After this I have to restart my daemon and then the CLI responds again but the channel is still there.

debuglog:
`
2018-08-24T14:43:58.293Z lightning_gossipd(20267): UPDATE WIRE_GOSSIP_GET_TXOUT

2018-08-24T14:43:58.306Z lightning_gossipd(20267): Received channel_update for 518494:2113:1 with old time 1526502819

2018-08-24T14:43:58.321Z lightning_gossipd(20267): Received channel_announcement for channel 518510:1692:0

2018-08-24T14:43:58.333Z lightningd(20257):jcon fd 22: Failing close: Channel is in state AWAITING_UNILATERAL
`
Any tips?

cdecker · 2018-08-24T21:10:51Z

Can you share the output of sqlite3 $HOME/.lightning/lightningd.sqlite3 "select hex(rawtx) from transactions"? That may contain the close transaction, which we can then broadcast again.

martijnbolt · 2018-08-24T21:56:04Z

thanks! is this what you're after?

root@retropie:~/.lightning# sqlite3 lightningd.sqlite3 "select hex(rawtx) from transactions"
02000000000101FE5B73607D2D9A2BA50A9A6B18D9BF5AB26F61EBF164E1E2C790E8D910A1C57D010000000040993A8000040047304402203D64688221EAC6C2E22487980015EF95C5D39C1BCE35895EA5FE9627E4D0D5D302204F592F9AE7123EE694BA488B015F72861C4DE1B26945753B9094741F404136C5014730440220303B779DE86407A2BBA60235FFDB1BF3102047D7BD6A05E95233F2C0CE49E9BC02202C22978648D6B7C060112D7B488B205DADAD5FD3949CC4CEC07DC723A541C7DE01475221021941C80DD3899CACF8A2171385FC0129C8B6EE1E34DDAC3FEBB0AF84C3F2372D2102EBA9C89D4C3A6A18B9D8B6129A900B788876EF5D61B5BE0E4291AEDD9608649B52AE8CA4B320
02000000000101022A66C4A878FF332FD7EAD8990064782EA8BEF6ACAEC8EBB5B7175F39B6897A0000000000FFFFFFFF0136850100000000001600149F51114DA9CB090643EAE750D335061A5484E170040047304402201C7499EF747278DA2332BDE7C11FF6B7C86031945BB27A57D4BC8BEAA1C1CC8C0220499B2D53BF399D1AD9F6D7F683200C9EC09DC84E8A59FBB1E6564DC5878B1A0601483045022100C88ED13371918D922C377C248E6F2FF3559D8453450338C11C41C51812D4C0FB02201A05DAA86E43E578B731C4D8F7E3ABABA0E54E3E887C22118422437A07A5B9410147522102B09C2BBE3EC738AF90E788051A2B55DF702A5C679E154CEF6868249558E5887721034FDE3C5968D03C9AC5AD0241B719B096CA10EE7AFC736FA150162AB22F782CBC52AE00000000

cdecker · 2018-08-25T12:47:16Z

Ok, the first transaction is what I was looking for:

{
  "txid": "229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c",
  "hash": "cc3dbb6e32679b0993cd41b572f5e166f010b77ad8469498bda55ccaa09f0230",
  "version": 2,
  "size": 271,
  "vsize": 106,
  "locktime": 548643980,
  "vin": [
    {
      "txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
      "vout": 1,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c501",
        "30440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01",
        "5221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae"
      ],
      "sequence": 2151323968
    }
  ],
  "vout": [
  ]
}

The part that is annoying here is that the closing transaction has no output. This is a known (and fixed) issue where the fee would dominate the entire value and no funds would actually be sent back, however a 0 output transaction is invalid to bitcoin clients, and will therefore never confirm.

The version you have there (b8c6365) is very old, that's why it's still producing these close transactions.

We have two options here:

Upgrade to a newer version, reset the channel state and try to close again. This either returns a minimal amount to you, or it spends everything in fees creating a single op-return output.
Use dev-forget-channel to simply forget about the channel and not attempt to recover the $1.34, but at least clear the channel from listpeers

cdecker · 2018-08-25T12:48:46Z

For reference: the commit that fixed this is 45e145d

martijnbolt · 2018-08-28T10:22:04Z

For some reason this does not work.

    sudo /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c
    { "code" : -32601, "message" : "Unknown command 'dev-forget-channel'" }

martijnbolt · 2018-08-29T06:51:30Z

I have installed and use a new version now. How do i reset the channel state or how do i fail the channel?

martijnbolt · 2018-08-29T07:26:27Z

I found another problem on my node, regarding a different much more recent channel. Now using the latest version of clightning:

pi@retropie:~ $ sudo /home/pi/lightning/cli/./lightning-cli connect 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266 23.237.77.11 9735
{
      "id": "0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266"
}
pi@retropie:~ $ sudo /home/pi/lightning/cli/./lightning-cli fundchannel 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266 277288
{ "code" : -1, "message" : "Peer already AWAITING_UNILATERAL" }

It seems multiple channels have the problem on my pi. How to get rid of these failed channels as dev-forget-channel does not seem to be available in v0.6-392-g898655f? (with or without recovering the funds)

cdecker · 2018-08-31T10:21:06Z

The dev-forget-channel command is a developer command, which can be enabled by recompiling with developer mode turned on:

./configure --enable-developer --disable-valgrind
make

Then restart the node and you should have the dev-forget-channel available.

martijnbolt · 2018-09-03T13:06:16Z

Thanks. I have tried this but still no luck unfortunately. After I check the results the channel is still there. As the command suggests using 'close' crashes clighting:

pi@retropie:/etc/systemd/system $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{ "code" : -1, "message" : "This channel has HTLCs attached and it is not safe to forget it. Please use `close` or `dev-fail` instead." }
pi@retropie:/etc/systemd/system $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel force='true' 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
lightning-cli: Expected key=value in '02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432': Success
pi@retropie:/etc/systemd/system $  /home/pi/clightning/lightning/cli/./lightning-cli listpeers | more
{
    "peers": [
    {
      "id": "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432",
      "connected": false,
      "channels": [
        {
          "state": "AWAITING_UNILATERAL",
          "short_channel_id": "506902:2127:1",
          "channel_id": "fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c",
          "funding_txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
          "msatoshi_to_us": 20000000,
          "msatoshi_to_us_min": 20000000,
          "msatoshi_to_us_max": 20000000,
          "msatoshi_total": 20000000, etc etc etc etc etc

ZmnSCPxj · 2018-09-03T13:41:09Z

Can you give result of close command?

martijnbolt · 2018-09-03T16:08:27Z

    pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close force='true' 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
    lightning-cli: Expected key=value in '02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432': Success
    pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close force=true 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
    lightning-cli: Expected key=value in '02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432': Success
    pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432

This last command hangs clightningd with this in the log:

    2018-09-03T15:54:53.923Z lightningd(15702):jcon fd 22: Failing close: Channel is in state AWAITING_UNILATERAL

Or it goes like this:

    pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
    { "code" : -1, "message" : "This channel has HTLCs attached and it is not safe to forget it. Please use `close` or `dev-fail` instead." }
    pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
    { "code" : -1, "message" : "Channel is in state AWAITING_UNILATERAL" }

With the following in the log:

2018-09-03T16:06:39.887Z lightningd(16156): 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432 chan #1: Peer permanent failure in AWAITING_UNILATERAL: Offered HTLC 0 SENT_ADD_ACK_REVOCATION cltv 507601 hit deadline
2018-09-03T16:06:39.887Z lightningd(16156):      (tx 229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c)
2018-09-03T16:06:39.891Z lightningd(16156): sendrawtransaction: 02000000000101fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57d010000000040993a8000040047304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c5014730440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01475221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae8ca4b320
2018-09-03T16:06:39.896Z lightningd(16156): FATAL SIGNAL 11 (version v0.6-392-g898655f)

ZmnSCPxj · 2018-09-04T12:23:00Z

   pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close force='true' 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
   lightning-cli: Expected key=value in '02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432': Success

The above is because you should use force=true and then id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432

lightning-cli close force=true id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432

If you will use key=value form, you must give it for all parameters.

The crash on close is concerning. In which version does it crash (it seems you have run multiple versions already at this point)?

cdecker · 2018-09-05T10:59:55Z

I think you will also need to specify that you're using the key-value format by having the first cli argument be -k like this:

lightning-cli close -k force=true id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432

martijnbolt · 2018-09-05T22:34:48Z

Not the result we're looking for unfortunately.

pi@retropie:~ $  /home/pi/clightning/lightning/cli/./lightning-cli close -k force=true id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{ "code" : -1, "message" : "Channel is in state AWAITING_UNILATERAL" }

with some logs:

2018-09-05T22:31:04.323Z lightningd(16222): FATAL SIGNAL 11 (version v0.6-392-g898655f)
2018-09-05T22:32:17.401Z lightning_gossipd(23444): gossip_store: Bad node_announcement (1002009501012b122f7d9e2a1f9c70f4feabcdb72d83f9d0de78841be09aef4662854dfca5261bdf9698f6c0bbe2a45013486cf770ca2c3a8755eb40b7cc71dd834f781e292d00005b5b748202b42e0c088aa1fc303aa18325afa23347d399b9e4cd0e77556ed0bcf4106e461980008047726170654a6f6c6c7952616e636865720000000000000000000000000000000007016cd018382607) truncating to 1
2018-09-05T22:32:20.171Z lightningd(23433): 024655b768ef40951b20053a5c4b951606d4d86085d51238f2c67c7dec29c792ca chan #8: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: received ERROR channel 0138202e043664c84a6a29b77ee50086a4eed2fec38ad97ed65ab13b626546c9: Can't find HTLC 7 to resend
2018-09-05T22:32:20.171Z lightningd(23433):      (tx 39376436d91cf873bf028737334af3d84b180b9f847e26115bea375f228bdd2f)
2018-09-05T22:32:37.847Z lightningd(23433): 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432 chan #1: Peer permanent failure in AWAITING_UNILATERAL: Offered HTLC 0 SENT_ADD_ACK_REVOCATION cltv 507601 hit deadline
2018-09-05T22:32:37.847Z lightningd(23433):      (tx 229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c)
2018-09-05T22:32:37.931Z lightningd(23433): Broadcasting tx 020000000001010138202e043664c84a6a29b77ee50086a4eed2fec38ad97ed65ab13b626546c9000000000068c7db80026a69000000000000160014f5b906a644b549281c61e8edfc3ecc0d7b2e8993e37b0000000000002200204200db4b27794064399940cacc736fdd34c21dc54eee32b025a57660f05aa29e0400483045022100f2ac84fea3d1dd7caac18e075c03ca44fbba606c887581f39daf98c51d7a058e02207f18f7f8af875733022fb241e64675c919527479d276abff08ac73d43206d730014830450221008cd3e4105bd617d4c10fd783c27673b105b2e880790fd8f97ad8e54e8618e0d002207d60f8691287125dfcabf4d7ceaa68080312518678861df6f81c42e13552a3950147522102b1e40a14bc810903a136ca0c7f45165662df01823fc75d23e471e0a7f9a99a4f21031ab28572ea083ba860fc02cb02b5f8726eec10a0a2f84caf20b6f7a684c01c9352ae05b0a320: 26 error code: -26?error message:?66: insufficient fee?

And a different try:

pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli close id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432

Results in a crash with no unusual logs.

The command dev-forget-channel also doesn't help out here:

pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{ "code" : -1, "message" : "This channel has HTLCs attached and it is not safe to forget it. Please use `close` or `dev-fail` instead." }
pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel force=true id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{ "code" : -1, "message" : "This channel has HTLCs attached and it is not safe to forget it. Please use `close` or `dev-fail` instead." }
pi@retropie:~ $ /home/pi/clightning/lightning/cli/./lightning-cli dev-fail id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{
}

The channel is still there. Logs:

2018-09-05T22:47:45.664Z lightningd(23559): 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432 chan #1: Peer internal error AWAITING_UNILATERAL: Failing due to dev-fail command
2018-09-05T22:47:45.665Z lightningd(23559): 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432 chan #1: Peer permanent failure in AWAITING_UNILATERAL: Internal error: Failing due to dev-fail command
2018-09-05T22:47:45.665Z lightningd(23559):      (tx 229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c)

I'm out of ideas here. Unless his has something to do with it:

2018-09-05T22:48:38.162Z lightningd(23559): Broadcasting tx 020000000001010138202e043664c84a6a29b77ee50086a4eed2fec38ad97ed65ab13b626546c9000000000068c7db80026a69000000000000160014f5b906a644b549281c61e8edfc3ecc0d7b2e8993e37b0000000000002200204200db4b27794064399940cacc736fdd34c21dc54eee32b025a57660f05aa29e0400483045022100f2ac84fea3d1dd7caac18e075c03ca44fbba606c887581f39daf98c51d7a058e02207f18f7f8af875733022fb241e64675c919527479d276abff08ac73d43206d730014830450221008cd3e4105bd617d4c10fd783c27673b105b2e880790fd8f97ad8e54e8618e0d002207d60f8691287125dfcabf4d7ceaa68080312518678861df6f81c42e13552a3950147522102b1e40a14bc810903a136ca0c7f45165662df01823fc75d23e471e0a7f9a99a4f21031ab28572ea083ba860fc02cb02b5f8726eec10a0a2f84caf20b6f7a684c01c9352ae05b0a320: 25 error code: -25?error message:?Missing inputs?
2018-09-05T22:48:38.172Z lightningd(23559): Broadcasting tx 02000000000101fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57d010000000040993a8000040047304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c5014730440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01475221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae8ca4b320: 26 error code: -26?error message:?16: bad-txns-vout-empty?
2018-09-05T22:48:38.182Z lightningd(23559): Broadcasting tx 02000000000101fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57d010000000040993a8000040047304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c5014730440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01475221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae8ca4b320: 26 error code: -26?error message:?16: bad-txns-vout-empty?
2018-09-05T22:48:38.192Z lightningd(23559): Broadcasting tx 02000000000101fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57d010000000040993a8000040047304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c5014730440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01475221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae8ca4b320: 26 error code: -26?error message:?16: bad-txns-vout-empty?

cdecker · 2018-09-06T12:51:12Z

OK, I see, the channels that give you bad-txns-vout-empty will indeed never close and you won't get funds back from them either, since the full balance went to fees (and we were producing empty transactions in that case).

I don't think we're going to be able to fix this... I checked on the remote end as well, and it also has the botched close transaction. The only thing I can offer is to hack the dev-forget-channel command to ignore any attached HTLCs.

martijnbolt · 2018-09-06T17:37:34Z

Thanks. That's good enough.

cdecker · 2018-09-07T10:42:20Z

Ok, if you remove these lines and recompile you should be able to forget with the force option:

lightning/lightningd/peer_control.c

Lines 1244 to 1252 in f2e0857

    
           if (channel_has_htlc_out(forget->channel) || 
        
               channel_has_htlc_in(forget->channel)) { 
        
           	command_fail(cmd, LIGHTNINGD, 
        
           		     "This channel has HTLCs attached and it is " 
        
           		     "not safe to forget it. Please use `close` " 
        
           		     "or `dev-fail` instead."); 
        
           	return; 
        
           }

martijnbolt · 2018-09-07T13:24:06Z

I have commented out those lines in the source and recompiled.

pi@retropie:~/clightning/lightning/lightningd $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel force=true id=02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
lightning-cli: reading response: Success

results in a crash:

2018-09-07T13:19:37.928Z lightningd(24980): 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432 chan #1: Peer permanent failure in AWAITING_UNILATERAL: Offered HTLC 0 SENT_ADD_ACK_REVOCATION cltv 507601 hit deadline
2018-09-07T13:19:37.928Z lightningd(24980):      (tx 229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c)
2018-09-07T13:20:12.274Z lightningd(24980): Freeing channel AWAITING_UNILATERAL has hout SENT_ADD_ACK_REVOCATION
2018-09-07T13:20:12.275Z lightningd(24980): FATAL SIGNAL 6 (version v0.6-392-g898655f-modded)

With the peer still being there:

pi@retropie:~/clightning/lightning/lightningd $ /home/pi/clightning/lightning/cli/./lightning-cli listpeers | jq [.]
[
{
"peers": [
  {
    "id": "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432",
    "connected": false,
    "channels": [
      {
        "state": "AWAITING_UNILATERAL",
        "short_channel_id": "506902:2127:1",
        "channel_id": "fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c",
        "funding_txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
        "msatoshi_to_us": 20000000,

cdecker · 2018-09-09T20:57:37Z

Oh, I missed that assertion. You also need to comment out these lines:

lightning/lightningd/channel.c

Lines 87 to 95 in 634f19a

    
           if (hout) 
        
           	fatal("Freeing channel %s has hout %s", 
        
           	      channel_state_name(channel), 
        
           	      htlc_state_name(hout->hstate)); 
        
           if (hin) 
        
           	fatal("Freeing channel %s has hin %s", 
        
           	      channel_state_name(channel), 
        
           	      htlc_state_name(hin->hstate));

martijnbolt · 2018-09-09T22:34:07Z

Thanks a lot for all the help. This works. :-)

cdecker · 2018-09-10T02:27:22Z

Glad we could get this cleaned up, and sorry for the hoops you had to jump through, but the code that was causing this has been fixed for a while now.

cdecker added the closingd label Aug 23, 2018

cdecker added the state::needinfo label Aug 23, 2018

cdecker mentioned this issue Aug 31, 2018

Can't close channels #1894

Closed

cdecker added state::stale This issue/PR has not seen any activity for a while, and has no actionable step. Will be closed soon and removed closingd labels Sep 6, 2018

cdecker closed this as completed Sep 10, 2018

Eskyee mentioned this issue May 14, 2019

Bitcoin c-lightning Channel "AWAITING_UNILATERAL" Peer connection lost closing, CPU 100% btcpayserver/btcpayserver#839

Closed

Eskyee mentioned this issue May 28, 2019

AWAITING_UNILATERAL bug #2684

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

can't close channel in state "CHANNELD_SHUTTING_DOWN" #1865

can't close channel in state "CHANNELD_SHUTTING_DOWN" #1865

martijnbolt commented Aug 22, 2018

cdecker commented Aug 23, 2018

martijnbolt commented Aug 24, 2018 •

edited

Loading

martijnbolt commented Aug 24, 2018 •

edited

Loading

cdecker commented Aug 24, 2018

martijnbolt commented Aug 24, 2018 •

edited by cdecker

Loading

cdecker commented Aug 25, 2018

cdecker commented Aug 25, 2018

martijnbolt commented Aug 28, 2018 •

edited

Loading

martijnbolt commented Aug 29, 2018 •

edited

Loading

martijnbolt commented Aug 29, 2018 •

edited

Loading

cdecker commented Aug 31, 2018

martijnbolt commented Sep 3, 2018 •

edited

Loading

ZmnSCPxj commented Sep 3, 2018

martijnbolt commented Sep 3, 2018 •

edited

Loading

ZmnSCPxj commented Sep 4, 2018 •

edited

Loading

cdecker commented Sep 5, 2018

martijnbolt commented Sep 5, 2018 •

edited

Loading

cdecker commented Sep 6, 2018

martijnbolt commented Sep 6, 2018

cdecker commented Sep 7, 2018

martijnbolt commented Sep 7, 2018 •

edited

Loading

cdecker commented Sep 9, 2018

martijnbolt commented Sep 9, 2018

cdecker commented Sep 10, 2018

can't close channel in state "CHANNELD_SHUTTING_DOWN" #1865

can't close channel in state "CHANNELD_SHUTTING_DOWN" #1865

Comments

martijnbolt commented Aug 22, 2018

Issue and Steps to Reproduce

getinfo output

cdecker commented Aug 23, 2018

martijnbolt commented Aug 24, 2018 • edited Loading

martijnbolt commented Aug 24, 2018 • edited Loading

cdecker commented Aug 24, 2018

martijnbolt commented Aug 24, 2018 • edited by cdecker Loading

cdecker commented Aug 25, 2018

cdecker commented Aug 25, 2018

martijnbolt commented Aug 28, 2018 • edited Loading

martijnbolt commented Aug 29, 2018 • edited Loading

martijnbolt commented Aug 29, 2018 • edited Loading

cdecker commented Aug 31, 2018

martijnbolt commented Sep 3, 2018 • edited Loading

ZmnSCPxj commented Sep 3, 2018

martijnbolt commented Sep 3, 2018 • edited Loading

ZmnSCPxj commented Sep 4, 2018 • edited Loading

cdecker commented Sep 5, 2018

martijnbolt commented Sep 5, 2018 • edited Loading

cdecker commented Sep 6, 2018

martijnbolt commented Sep 6, 2018

cdecker commented Sep 7, 2018

martijnbolt commented Sep 7, 2018 • edited Loading

cdecker commented Sep 9, 2018

martijnbolt commented Sep 9, 2018

cdecker commented Sep 10, 2018

`getinfo` output

martijnbolt commented Aug 24, 2018 •

edited

Loading

martijnbolt commented Aug 24, 2018 •

edited

Loading

martijnbolt commented Aug 24, 2018 •

edited by cdecker

Loading

martijnbolt commented Aug 28, 2018 •

edited

Loading

martijnbolt commented Aug 29, 2018 •

edited

Loading

martijnbolt commented Aug 29, 2018 •

edited

Loading

martijnbolt commented Sep 3, 2018 •

edited

Loading

martijnbolt commented Sep 3, 2018 •

edited

Loading

ZmnSCPxj commented Sep 4, 2018 •

edited

Loading

martijnbolt commented Sep 5, 2018 •

edited

Loading

martijnbolt commented Sep 7, 2018 •

edited

Loading