Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash in connectd due to bad gossip_store #5572

Closed
rustyrussell opened this issue Sep 5, 2022 · 0 comments
Closed

crash in connectd due to bad gossip_store #5572

rustyrussell opened this issue Sep 5, 2022 · 0 comments

Comments

@rustyrussell
Copy link
Contributor

We hit an assert as we try to send a giant status message (> 128MB), but the reason we're sending it is because a gossip_store entry fails checksum. I accessed the gossip_store shortly after this, and it seemed fine.

The most likely explanation is that connectd somehow got a bogus offset, and was reading junk from the store.

lightning_connectd: FATAL SIGNAL (version v0.12.0)
0x414a56 send_backtrace
        common/daemon.c:33
0x41d10a status_failed
        common/status.c:221
0x41d1fc status_backtrace_exit
        common/subdaemon.c:18
0x414ada crashdump
        common/daemon.c:49
0x7f2a8063c31f ???
        ???:0
0x7f2a8063c2a2 ???
        ???:0
0x7f2a806258a3 ???
        ???:0
0x7f2a80625788 ???
        ???:0
0x7f2a80634a15 ???
        ???:0
0x42d90d wire_sync_write
        wire/wire_sync.c:13
0x414d75 daemon_conn_sync_flush
        common/daemon_conn.c:114
0x41cb31 flush_and_exit
        common/status.c:189
0x41d01a status_send_fatal
        common/status.c:206
0x41d102 status_failed
        common/status.c:223
0x416f16 gossip_store_next
        common/gossip_store.c:156
0x40d138 maybe_from_gossip_store
        connectd/multiplex.c:530
0x40d3fe write_to_peer
        connectd/multiplex.c:966
0x4aee10 next_plan
        ccan/ccan/io/io.c:59
0x4af210 io_do_always
        ccan/ccan/io/io.c:435
0x4aff02 handle_always
        ccan/ccan/io/poll.c:304
0x4b0229 io_loop
        ccan/ccan/io/poll.c:385
0x40b1db main
        connectd/connectd.c:2063
0x7f2a80626b74 ???
        ???:0
0x40481d ???
        ???:0
0xffffffffffffffff ???
        ???:0
lightning_connectd: wire/wire_sync.c:13: wire_sync_write: Assertion `tal_bytelen(msg) < WIRE_LEN_LIMIT' failed.
Lost connection to the RPC socket.Lost connection to the RPC socket.
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Sep 5, 2022
Should help diagnose ElementsProject#5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
niftynei pushed a commit that referenced this issue Sep 7, 2022
Should help diagnose #5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
endothermicdev added a commit that referenced this issue Sep 9, 2022
Working on diagnosing #5565, #5572 and preventing a crash.
endothermicdev added a commit that referenced this issue Sep 9, 2022
niftynei pushed a commit to niftynei/lightning that referenced this issue Sep 13, 2022
Should help diagnose ElementsProject#5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
niftynei pushed a commit to niftynei/lightning that referenced this issue Sep 13, 2022
Should help diagnose ElementsProject#5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
endothermicdev added a commit to endothermicdev/lightning that referenced this issue Sep 14, 2022
The gossip_store version byte was unaccounted for in the initial traversal
of gossip_store_end. This lead to an offset and a bogus message length
field. As a result, an early portion of the gossip_store could have been
skipped, potentially leading to gossip propagation issues downstream.

Fixes ElementsProject#5572 ElementsProject#5565

Changelog-fixed: proper gossip_store operation may resolve some previous gossip propagation issues
niftynei pushed a commit that referenced this issue Sep 14, 2022
The gossip_store version byte was unaccounted for in the initial traversal
of gossip_store_end. This lead to an offset and a bogus message length
field. As a result, an early portion of the gossip_store could have been
skipped, potentially leading to gossip propagation issues downstream.

Fixes #5572 #5565

Changelog-fixed: proper gossip_store operation may resolve some previous gossip propagation issues
whitslack pushed a commit to whitslack/lightning that referenced this issue Oct 8, 2022
Should help diagnose ElementsProject#5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
whitslack pushed a commit to whitslack/lightning that referenced this issue Oct 8, 2022
The gossip_store version byte was unaccounted for in the initial traversal
of gossip_store_end. This lead to an offset and a bogus message length
field. As a result, an early portion of the gossip_store could have been
skipped, potentially leading to gossip propagation issues downstream.

Fixes ElementsProject#5572 ElementsProject#5565

Changelog-fixed: proper gossip_store operation may resolve some previous gossip propagation issues
whitslack pushed a commit to whitslack/lightning that referenced this issue Oct 28, 2022
Should help diagnose ElementsProject#5572
which hit the invalid csum on a >64MB entry, if it happens again.

Signed-off-by: Rusty Russell <[email protected]>
whitslack pushed a commit to whitslack/lightning that referenced this issue Oct 28, 2022
The gossip_store version byte was unaccounted for in the initial traversal
of gossip_store_end. This lead to an offset and a bogus message length
field. As a result, an early portion of the gossip_store could have been
skipped, potentially leading to gossip propagation issues downstream.

Fixes ElementsProject#5572 ElementsProject#5565

Changelog-fixed: proper gossip_store operation may resolve some previous gossip propagation issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant