Jasan/guilt configure rebased ammended #3

jsarenik · 2018-06-06T13:34:08Z

This PR is already rebased to current master.

We generate new ones anyway; removing this code changes fixes coming up which now only need to change one place. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

…haviour. The gossip-query spec enhancements say not to forward an channel_announcement until you have receive a channel_update. This test fails for now. Signed-off-by: Rusty Russell <[email protected]>

…pdate. Signed-off-by: Rusty Russell <[email protected]>

…roadcast. handle_pending_cannouncement might not actually add the announcment, as it could be waiting for a channel_update. We need to wait for the actual announcement before considering announcing our node. Signed-off-by: Rusty Russell <[email protected]>

In general, we need to only publish node announcements after publishing channel announcements, though we can accept node announcements as soon as we see channel announcements. So we keep a flag for those node_announcement which haven't been broadcast yet. Signed-off-by: Rusty Russell <[email protected]>

From BOLT #9 proposed update. Signed-off-by: Rusty Russell <[email protected]>

This includes the gossip query messages. Signed-off-by: Rusty Russell <[email protected]>

We use the same system as for gossip: we trickle out replies when we're otherwise idle. This is minimal infrastructure: we don't actually process the query_short_channel_ids message yet, nor do we append node announcements. Signed-off-by: Rusty Russell <[email protected]>

We use the same system as for gossip: we trickle out replies when we're otherwise idle. As we trickle out replies to query_short_channel_ids, we remember the pubkeys of nodes we mention. At the end, we sort and uniquify, and then send any node_announcements we have for those. Signed-off-by: Rusty Russell <[email protected]>

This doesn't handle zlib yet. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

And write the test for it. Signed-off-by: Rusty Russell <[email protected]>

This lets us filter by timestamp. Signed-off-by: Rusty Russell <[email protected]>

This is kind of orthogonal to the other changes, but makes sense: if we would instantly or never prune the message, don't accept it. Signed-off-by: Rusty Russell <[email protected]>

And initialize filter (to "never") when we negotiated LOCAL_GOSSIP_QUERIES, and send initial filter message. Signed-off-by: Rusty Russell <[email protected]>

We have a function called 'wake_pkt_out' which is really 'start gossiping', so rename it to 'wake_gossip_out'. In addition, it's fired both on a timer, and in response to our first gossip_timestamp_filter, which leads to very confusing (though, technically, not incorrect) behavior. Keep a single timer at all times, which now doubles as the flag to indicating we're syncing right now. Set it once we're done syncing gossip. Technically this means we got from once-every-60-seconds to quiet-for-60-seconds-between-gossip, but that's OK. Signed-off-by: Rusty Russell <[email protected]>

…filtering. Since we currently only (ab)use it to send everything, we need a way to generate boutique queries for testing. Signed-off-by: Rusty Russell <[email protected]>

We send them all the short_channel_ids we have in a given range. Signed-off-by: Rusty Russell <[email protected]>

We keep a crappy bitmap, and finish when their replies cover everything we asked. Signed-off-by: Rusty Russell <[email protected]>

…eplies. We cap each reply at a single one, which forces the code into our recursion logic. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

We still use uncompressed if zlib turns out to be larger. Signed-off-by: Rusty Russell <[email protected]>

Previous replies weren't large enough; add another channel and then use IO tracing to make sure the reply is zlib encoded. Signed-off-by: Rusty Russell <[email protected]>

We *accept* a node_announce if we have a channel_announce, but we can't queue it until we queue the channel_announce, which we only do once we have recieved a channel_update. Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Rusty Russell <[email protected]>

@cdecker

Suggested-by: @cdecker Signed-off-by: Rusty Russell <[email protected]>

The whole test is under DEVELOPER anyway, but even if it weren't it would work fine. Signed-off-by: Rusty Russell <[email protected]>

@cdecker

Suggested-by: @cdecker Signed-off-by: Rusty Russell <[email protected]>

Signed-off-by: Christian Decker <[email protected]>

Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e) #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19 #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226 #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331 #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113 #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39 #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382 #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466 #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228 #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474 #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59 #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407 #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417 #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445 #14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24 ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822 ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Direct leak of 1024 byte(s) in 2 object(s) allocated from: #0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448) #1 0x55d11b782c96 in timer_default_alloc ccan/ccan/timer/timer.c:16 #2 0x55d11b7832b7 in add_level ccan/ccan/timer/timer.c:166 #3 0x55d11b783864 in timer_fast_forward ccan/ccan/timer/timer.c:334 #4 0x55d11b78396a in timers_expire ccan/ccan/timer/timer.c:359 #5 0x55d11b774993 in io_loop ccan/ccan/io/poll.c:395 #6 0x55d11b72322f in plugins_init lightningd/plugin.c:1013 #7 0x55d11b7060ea in main lightningd/lightningd.c:664 #8 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Indirect leak of 48 byte(s) in 1 object(s) allocated from: #0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448) #1 0x55d11b77d270 in strmap_add_ ccan/ccan/strmap/strmap.c:90 #2 0x55d11b704603 in command_set_usage lightningd/jsonrpc.c:891 #3 0x55d11b733cb5 in param common/param.c:295 #4 0x55d11b6f7b37 in json_connect lightningd/connect_control.c:96 #5 0x55d11b7042ef in setup_command_usage lightningd/jsonrpc.c:841 #6 0x55d11b70443b in jsonrpc_command_add_perm lightningd/jsonrpc.c:863 #7 0x55d11b704533 in jsonrpc_setup lightningd/jsonrpc.c:876 #8 0x55d11b705695 in new_lightningd lightningd/lightningd.c:210 #9 0x55d11b706062 in main lightningd/lightningd.c:644 #10 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e) #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19 #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226 #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331 #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83 #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626 #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694 #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466 #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214 #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244 #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250 #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235 #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397 #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481 #14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450 ElementsProject#15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449 ElementsProject#16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24 ElementsProject#17 0x564ee89ba540 in main lightningd/lightningd.c:822 ElementsProject#18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Direct leak of 32 byte(s) in 1 object(s) allocated from: #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e) #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19 #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226 #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331 #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30 #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77 #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91 #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482 #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721 #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e) #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19 #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226 #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331 #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61 #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116 #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365 #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603 #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620 #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694 #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466 #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214 #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244 #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250 #14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235 ElementsProject#15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397 ElementsProject#16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481 ElementsProject#17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450 ElementsProject#18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449 ElementsProject#19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24 ElementsProject#20 0x55cac1e0156b in main lightningd/lightningd.c:822 ElementsProject#21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e) #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19 #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226 #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331 #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113 #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39 #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382 #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466 #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228 #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474 #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59 #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407 #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417 #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445 #14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24 ElementsProject#15 0x555ce2a6e56b in main lightningd/lightningd.c:822 ElementsProject#16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) Signed-off-by: Rusty Russell <[email protected]>

Direct leak of 1024 byte(s) in 2 object(s) allocated from: #0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448) #1 0x55d11b782c96 in timer_default_alloc ccan/ccan/timer/timer.c:16 #2 0x55d11b7832b7 in add_level ccan/ccan/timer/timer.c:166 #3 0x55d11b783864 in timer_fast_forward ccan/ccan/timer/timer.c:334 #4 0x55d11b78396a in timers_expire ccan/ccan/timer/timer.c:359 #5 0x55d11b774993 in io_loop ccan/ccan/io/poll.c:395 #6 0x55d11b72322f in plugins_init lightningd/plugin.c:1013 #7 0x55d11b7060ea in main lightningd/lightningd.c:664 #8 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a) To fix this, we actually make 'ld->timers' a pointer, so we can clean it up last of all. We can't free it before ld, because that causes timers to be destroyed. Signed-off-by: Rusty Russell <[email protected]>