Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Funding channel causes crash due to "UNIQUE constraint failed: peers.node_id" #1029

Closed
sadilek opened this issue Feb 19, 2018 · 12 comments
Closed

Comments

@sadilek
Copy link

sadilek commented Feb 19, 2018

I connected to a node and opened a channel:

$ lightning-cli listpeers | grep 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac
<empty, not connected to this node yet>
$ lightning-cli connect 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac 52.215.249.189 9735
{ "id" : "0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac" }
$ lightning-cli fundchannel 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac 50000
lightning-cli: reading response: Success

Lightningd crashed:

2018-02-18T23:43:21.033Z lightningd(17531): 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac chan #0: State changed from UNINITIALIZED to OPENINGD
wallet_channel_save: UNIQUE constraint failed: peers.node_id
2018-02-18T23:43:21.040Z lightningd(17531): wallet_channel_save: UNIQUE constraint failed: peers.node_id
Log dumped in crash.log
2018-02-18T23:43:21.849Z lightningd(17531): FATAL SIGNAL 6 RECEIVED
2018-02-18T23:43:21.861Z lightningd(17531): backtrace: lightningd/log.c:498 (log_crash) 0x411f92
2018-02-18T23:43:21.861Z lightningd(17531): backtrace: (null):0 ((null)) 0x7f351c4694af
2018-02-18T23:43:21.861Z lightningd(17531): backtrace: (null):0 ((null)) 0x7f351c469428
2018-02-18T23:43:21.861Z lightningd(17531): backtrace: (null):0 ((null)) 0x7f351c46b029
2018-02-18T23:43:21.861Z lightningd(17531): backtrace: lightningd/log.c:587 (fatal) 0x4123cc
2018-02-18T23:43:21.862Z lightningd(17531): backtrace: wallet/db.c:217 (db_exec_prepared_) 0x44e5dc
2018-02-18T23:43:21.862Z lightningd(17531): backtrace: wallet/wallet.c:800 (wallet_channel_save) 0x4535d3
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: lightningd/peer_control.c:2415 (peer_offer_channel) 0x420902
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: lightningd/peer_control.c:2492 (gossip_peer_released) 0x420d73
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: lightningd/subd.c:306 (sd_msg_reply) 0x4269e0
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: lightningd/subd.c:454 (sd_msg_read) 0x426fc5
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: lightningd/subd.c:327 (read_fds) 0x426add
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x45e440
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: ccan/ccan/io/io.c:387 (do_plan) 0x45ef17
2018-02-18T23:43:21.863Z lightningd(17531): backtrace: ccan/ccan/io/io.c:397 (io_ready) 0x45ef55
2018-02-18T23:43:21.864Z lightningd(17531): backtrace: ccan/ccan/io/poll.c:310 (io_loop) 0x460824
2018-02-18T23:43:21.864Z lightningd(17531): backtrace: lightningd/lightningd.c:378 (main) 0x4106e6
2018-02-18T23:43:21.864Z lightningd(17531): backtrace: (null):0 ((null)) 0x7f351c45482f
2018-02-18T23:43:21.864Z lightningd(17531): backtrace: (null):0 ((null)) 0x4032d8
2018-02-18T23:43:21.864Z lightningd(17531): backtrace: (null):0 ((null)) 0xffffffffffffffff
Fatal signal 6. 0x412078 log_crash
        lightningd/log.c:524
0x7f351c4694af ???
        ???:0
0x7f351c469428 ???
        ???:0
0x7f351c46b029 ???
        ???:0
0x4123cc fatal
        lightningd/log.c:587
0x44e5dc db_exec_prepared_
        wallet/db.c:217
0x4535d3 wallet_channel_save
        wallet/wallet.c:800
0x420902 peer_offer_channel
        lightningd/peer_control.c:2415
0x420d73 gossip_peer_released
        lightningd/peer_control.c:2492
0x4269e0 sd_msg_reply
        lightningd/subd.c:306
0x426fc5 sd_msg_read
        lightningd/subd.c:454
0x426add read_fds
        lightningd/subd.c:327
0x45e440 next_plan
        ccan/ccan/io/io.c:59
0x45ef17 do_plan
        ccan/ccan/io/io.c:387
0x45ef55 io_ready
        ccan/ccan/io/io.c:397
0x460824 io_loop
        ccan/ccan/io/poll.c:310
0x4106e6 main
        lightningd/lightningd.c:378
0x7f351c45482f ???
        ???:0
0x4032d8 ???

By the time of the crash, my node was fully caught up. After the restart, it's now catching up again:

$ lightning-cli getinfo
{ "id" : "036487e0bd981617c674c2856fc0050472a0182838e0a4a1a67545c6cdee9f49a0", "port" : 9735, "address" : 
	[ 
		{ "type" : "ipv4", "address" : "81.6.40.158", "port" : 9735 } ], "version" : "v0.5.2-2016-11-21-1990-gb2819f9", "blockheight" : 507142, "network" : "bitcoin" }

I might have been connected to this node in the past. I happen to have a copy of the sqllite DB from before the crash.

$ sqlite3 /tmp/lightningd.sqlite3 
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> select id, hex(node_id), address from peers where hex(node_id) like "0333%";
102|0333B58F86A0C31DA7EE61A4BC2232FACCF8C4FE4167FE0DD9CE019FE03E82FAAC|52.215.249.189:9735

So despite listpeers not listing the node, it's still in the peers table. There is no channel referencing it though:
sqlite> select id, state from channels where hex(peer_id) like "0333%";

I can reproduce this, so let me know if you need more info.

@cdecker
Copy link
Member

cdecker commented Feb 19, 2018

Thanks @sadilek for the detailed issue report. I think this is similar to (if not a duplicate of) #1000

@sadilek
Copy link
Author

sadilek commented Feb 19, 2018

Indeed, sorry for missing that one. The issue seems to have been resolved for @Sjors after a dev-rescan-outputs. I can still reproduce it though. My node was caught up by now, and I ran this:

$ lightning-cli dev-rescan-outputs
<long output>
$ lightning-cli listpeers | grep 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac
$ sqlite3 .lightning/lightningd.sqlite3 
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> select id, hex(node_id), address from peers where hex(node_id) like "0333%";
102|0333B58F86A0C31DA7EE61A4BC2232FACCF8C4FE4167FE0DD9CE019FE03E82FAAC|52.215.249.189:9735
$ lightning-cli connect 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac 52.215.249.189 9735
{ "id" : "0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac" }
$ lightning-cli listpeers | grep 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac
		{ "state" : "GOSSIPING", "id" : "0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac", "netaddr" : 
$ lightning-cli fundchannel 0333b58f86a0c31da7ee61a4bc2232faccf8c4fe4167fe0dd9ce019fe03e82faac 50000
lightning-cli: reading response: Success

Node crashed with same error.

@rustyrussell
Copy link
Contributor

Hmm, what state does the db say it's in?

select state, id, hex(node_id), address from peers where hex(node_id) like "0333%";

@cdecker
Copy link
Member

cdecker commented Feb 20, 2018

@rustyrussell peers doesn't have a state, it's channels that have :-)

@sadilek
Copy link
Author

sadilek commented Feb 20, 2018

@rustyrussell, still the same: There is an entry for the peer but no channel for it:

sqlite> select id, hex(node_id), address from peers where hex(node_id) like "0333%";
102|0333B58F86A0C31DA7EE61A4BC2232FACCF8C4FE4167FE0DD9CE019FE03E82FAAC|52.215.249.189:9735
sqlite> select id, state from channels where hex(peer_id) like "0333%";
<empty>

@windsok
Copy link
Contributor

windsok commented Feb 21, 2018

I also have this error, running commit e48a97f

This occurred while the node was unattended, so was not due to any direct action by me

+266177.212285888 lightningd(29959):INFO: 0211ad124eaecaf2b223bad2f5381e81b3639f2a652b5aa7a0ceffc8593931fc30 chan #0: State changed from UNINITIALIZED to OPENINGD
+266177.222512343 lightningd(29959):DEBUG: lightning_openingd-0211ad124eaecaf2b223bad2f5381e81b3639f2a652b5aa7a0ceffc8593931fc30 chan #0: pid 19977, msgfd 16
+266177.231419094 lightningd(29959):BROKEN: wallet_channel_save: UNIQUE constraint failed: peers.node_id
+266177.555645182 lightningd(29959):BROKEN: FATAL SIGNAL 6 RECEIVED
+266177.575493838 lightningd(29959):BROKEN: backtrace: lightningd/log.c:498 (log_crash) 0x411c09
+266177.575731844 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0x7f1cc16ef4af
+266177.575859353 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0x7f1cc16ef428
+266177.575961162 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0x7f1cc16f1029
+266177.576056290 lightningd(29959):BROKEN: backtrace: lightningd/log.c:587 (fatal) 0x412043
+266177.576339176 lightningd(29959):BROKEN: backtrace: wallet/db.c:217 (db_exec_prepared_) 0x44d92e
+266177.576874386 lightningd(29959):BROKEN: backtrace: wallet/wallet.c:759 (wallet_channel_save) 0x452847
+266177.577675432 lightningd(29959):BROKEN: backtrace: lightningd/peer_control.c:2341 (peer_accept_channel) 0x41f92d
+266177.577886446 lightningd(29959):BROKEN: backtrace: lightningd/peer_control.c:506 (peer_sent_nongossip) 0x41a5d7
+266177.578221792 lightningd(29959):BROKEN: backtrace: lightningd/gossip_control.c:55 (peer_nongossip) 0x408d93
+266177.578343360 lightningd(29959):BROKEN: backtrace: lightningd/gossip_control.c:144 (gossip_msg) 0x408fed
+266177.578758345 lightningd(29959):BROKEN: backtrace: lightningd/subd.c:501 (sd_msg_read) 0x4264f3
+266177.578874828 lightningd(29959):BROKEN: backtrace: lightningd/subd.c:327 (read_fds) 0x425e32
+266177.579200967 lightningd(29959):BROKEN: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x45d44e
+266177.579371911 lightningd(29959):BROKEN: backtrace: ccan/ccan/io/io.c:387 (do_plan) 0x45df25
+266177.579471708 lightningd(29959):BROKEN: backtrace: ccan/ccan/io/io.c:397 (io_ready) 0x45df63
+266177.579698965 lightningd(29959):BROKEN: backtrace: ccan/ccan/io/poll.c:310 (io_loop) 0x45f832
+266177.579990511 lightningd(29959):BROKEN: backtrace: lightningd/lightningd.c:351 (main) 0x41035d
+266177.580097459 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0x7f1cc16da82f
+266177.580189200 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0x403248
+266177.580278106 lightningd(29959):BROKEN: backtrace: (null):0 ((null)) 0xffffffffffffffff

@windsok
Copy link
Contributor

windsok commented Feb 21, 2018

sqlite> select id, hex(node_id), address from peers where hex(node_id) like "0211ad%";
46|0211AD124EAECAF2B223BAD2F5381E81B3639F2A652B5AA7A0CEFFC8593931FC30|[::ffff:47.184.129.94]:48294
sqlite> select id, state from channels where hex(peer_id) like "0211ad%";
<empty>

@laanwj
Copy link
Contributor

laanwj commented Feb 21, 2018

Same issue here, it happens reproducibly between two of my own nodes, which used to have a channel in the past. This channel was auto-closed due to unknown (to me) reasons. When I try to fundchannel a new channel from the one to the other node, on the connected-to node I get the error:

wallet_channel_insert: UNIQUE constraint failed: peers.node_id

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f2cc7c7f3fa in __GI_abort () at abort.c:89
#2  0x0000556ecf2104ca in fatal (fmt=0x556ecf2aeea5 "%s: %s") at lightningd/log.c:592
#3  0x0000556ecf250fe1 in db_exec_prepared_ (caller=0x556ecf2b1ba0 <__func__.9071> "wallet_channel_insert", db=0x556ecf5a89c0, stmt=0x556ed0b292d8) at wallet/db.c:219
#4  0x0000556ecf2566ef in wallet_channel_insert (w=0x556ecf5a31a0, chan=0x556ed0b27770) at wallet/wallet.c:897
#5  0x0000556ecf21e298 in wallet_commit_channel (ld=0x556ecf5a2030, uc=0x556ed0b2fd80, remote_commit=0x556ed0b31100, remote_commit_sig=0x7fff34014870, funding_txid=0x7fff34014850, funding_outnum=1, 
    funding_satoshi=129845, push_msat=0, channel_flags=1 '\001', channel_info=0x7fff34014940, feerate=293) at lightningd/peer_control.c:2239
#6  0x0000556ecf21ee11 in opening_fundee_finished (openingd=0x556ed0b302f0, reply=0x556ed0b27260 <incomplete sequence \327>, fds=0x556ed0b306e0, uc=0x556ed0b2fd80) at lightningd/peer_control.c:2445
#7  0x0000556ecf225d10 in sd_msg_reply (conn=0x556ed0b305e0, sd=0x556ed0b302f0, sr=0x556ed0b30b50) at lightningd/subd.c:307
#8  0x0000556ecf2262cb in sd_msg_read (conn=0x556ed0b305e0, sd=0x556ed0b302f0) at lightningd/subd.c:435
#9  0x0000556ecf225e11 in read_fds (conn=0x556ed0b305e0, sd=0x556ed0b302f0) at lightningd/subd.c:328
#10 0x0000556ecf2615c5 in next_plan (conn=0x556ed0b305e0, plan=0x556ed0b30610) at ccan/ccan/io/io.c:59
#11 0x0000556ecf2620c2 in do_plan (conn=0x556ed0b305e0, plan=0x556ed0b30610, idle_on_epipe=false) at ccan/ccan/io/io.c:387
#12 0x0000556ecf262100 in io_ready (conn=0x556ed0b305e0, pollflags=1) at ccan/ccan/io/io.c:397
#13 0x0000556ecf263a4a in io_loop (timers=0x556ecf5a2120, expired=0x7fff34014d78) at ccan/ccan/io/poll.c:310
#14 0x0000556ecf20e71a in main (argc=9, argv=0x7fff34014ea8) at lightningd/lightningd.c:373

Same result as those above when trying with sqlite3: the peer is in the database, no channels.

rustyrussell added a commit to rustyrussell/lightning that referenced this issue Feb 22, 2018
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Feb 22, 2018
@rustyrussell
Copy link
Contributor

rustyrussell commented Feb 22, 2018 via email

cdecker pushed a commit that referenced this issue Feb 22, 2018
Fixes: #1029
Signed-off-by: Rusty Russell <[email protected]>
@sadilek
Copy link
Author

sadilek commented Feb 22, 2018

Thanks, I can confirm that this issue is fixed for me!

@yoss1x
Copy link

yoss1x commented Dec 23, 2018

I'm having the same issue with v0.6.2 - peers table has a record which no channel on channels table is referencing. When said peer is trying to fundchannel the remote node crashes:

2018-12-22T07:54:46.697Z lightningd(10338): wallet_channel_insert: UNIQUE constraint failed: peers.node_id
2018-12-22T07:54:46.721Z lightningd(10338): FATAL SIGNAL 6 (version v0.6.2)

Here are the logs from the channel closing event few weeks ago:

2018-11-26T18:23:58.125Z lightningd(13373): 0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got depth change 261->262 for 3c7b9328db3c207cfba20ef6331958b6d6b3bf2595e658db84cb451552da728c
2018-11-26T18:23:58.125Z lightningd(13373): 0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got depth change 260->261 for 915907e6af13e687a6f6e7ddf9ce4ec11edc4ddc5cb78288d22b6937170aae07
2018-11-26T18:23:58.125Z lightningd(13373): 0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got depth change 99->100 for d86d4ef96994b6742c5e6b001e32d96d189984548caca06ac4b3d83fa14c0bd1
2018-11-26T18:23:58.125Z lightningd(13373): 03e8e42ffb48766e0c7202b3ac53b6d5ce8348a5a39c0e42d4fac0a50d25a91411 chan #6266: Got depth change 1555->1556 for 8100f040639e2de0eaeb66dd087ac573bc4d1de3f57b535d371148f5aa0b43b3
2018-11-26T18:23:58.125Z lightningd(13373): 03e8e42ffb48766e0c7202b3ac53b6d5ce8348a5a39c0e42d4fac0a50d25a91411 chan #6266: Funding tx 8100f040639e2de0eaeb66dd087ac573bc4d1de3f57b535d371148f5aa0b43b3 depth 1556 of 3
2018-11-26T18:23:58.125Z lightningd(13373): 03e8e42ffb48766e0c7202b3ac53b6d5ce8348a5a39c0e42d4fac0a50d25a91411 chan #6266: Funding tx confirmed, but peer disconnected
2018-11-26T18:23:58.125Z lightningd(13373): 039199b7b5146fa2b0bd00a4c97ccbc4950c079ad5f9f982a8b8f2d8e00c828cd9 chan #5962: Got depth change 5681->5682 for 0eba92a77552e0c3a2538acbda37c1bdacf2dde384ecf9093187c980861068e7
2018-11-26T18:23:58.125Z lightningd(13373): 039199b7b5146fa2b0bd00a4c97ccbc4950c079ad5f9f982a8b8f2d8e00c828cd9 chan #5962: Funding tx 0eba92a77552e0c3a2538acbda37c1bdacf2dde384ecf9093187c980861068e7 depth 5682 of 3
2018-11-26T18:23:58.125Z lightningd(13373): 039199b7b5146fa2b0bd00a4c97ccbc4950c079ad5f9f982a8b8f2d8e00c828cd9 chan #5962: Funding tx confirmed, but peer disconnected
2018-11-26T18:23:58.126Z lightningd(13373): sendrawtransaction: 0200000000010114053fb5ff5e4c746454fec43d07e54e05e59a301bc6c145aa87afcdbab3e63e0100000000817508800108aa050000000000160014eff3e1e48c261294813fde1c2b721f560e9068ec0400473044022024305e8c646bff4b05d38a2e46bc7a2865c349c93f257cdec3d6ef5cacd0696d02204e86db932c1ca8ea15049a0a3c362b2a3590db724dd14bd494918d4d777414af01483045022100feab555277fb77dfd9f753ed34ff1de72cf45826ba57f0ed8a4fd6dcc35b9142022022932ab747c4950740be9f19e29debb3e7d78dd324af7be29e87fffbf01ef7030147522102ab659dd2392b22bc07f55d57e5df9dc11b3ffddc08966f10dfd6cd1169c68b5b2103cbefb28d61f490458ffe2459089e58399bef1ac6317944f48fc234e0e486b70952aec82ee220
2018-11-26T18:23:58.134Z lightningd(13373): sendrawtx exit 0, gave 0c711e6051e60d997f4ab09685610e57820c12428c8b7d79d357195920e6dec5?
2018-11-26T18:23:58.134Z lightningd(13373): l0394ightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got new message WIRE_ONCHAIN_DEPTH
2018-11-26T18:23:58.134Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: FUNDING_TRANSACTION/FUNDING_OUTPUT->OUR_UNILATERAL depth 262
2018-11-26T18:23:58.134Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: OUR_UNILATERAL/OUTPUT_TO_THEM->SELF depth 262
2018-11-26T18:23:58.135Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got new message WIRE_ONCHAIN_DEPTH
2018-11-26T18:23:58.135Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: OUR_UNILATERAL/THEIR_HTLC->OUR_HTLC_SUCCESS_TX depth 261
2018-11-26T18:23:58.135Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: Got new message WIRE_ONCHAIN_DEPTH
2018-11-26T18:23:58.135Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: OUR_HTLC_SUCCESS_TX/DELAYED_OUTPUT_TO_US->OUR_DELAYED_RETURN_TO_WALLET depth 100
2018-11-26T18:23:58.135Z lightningd(13373): lightning_onchaind-0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: UPDATE WIRE_ONCHAIN_ALL_IRREVOCABLY_RESOLVED
2018-11-26T18:23:58.135Z lightningd(13373): 0394494528c862b93b6b8caea83c5d73d5b767c9b48816178f8d0e3b54f54a46a5 chan #7940: onchaind complete, forgetting peer

While #1066 removes such channel-less peers during upgrade, until it does, a peer in that state will crash a node whenever it tries opening a channel.

Wasn't sure if to comment here or open new, let me know if a new issue is better.

@ZmnSCPxj
Copy link
Collaborator

@yoss1x maybe better to make a new issue, referencing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants