Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic: called fail_with on already-failed connection state #1510

Closed
rex4539 opened this issue Dec 13, 2020 · 7 comments · Fixed by #1531
Closed

Panic: called fail_with on already-failed connection state #1510

rex4539 opened this issue Dec 13, 2020 · 7 comments · Fixed by #1531
Assignees

Comments

@rex4539
Copy link
Contributor

rex4539 commented Dec 13, 2020

Suggested Solution

Stop calling fail_with on already-failed connection states.

Detailed Logs

Click triangle to reveal logs

Dec 13 12:39:40.386  INFO {zebrad="83e72bdd"}:peer{addr=[2001:19f0:5001:d0a:5400:1ff:fed2:7142]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(550808) response_len=158 chain_tip_height=Height(597795) stop_height=None intersection_height=Some(Height(550650))
Dec 13 12:39:41.022  WARN {zebrad="83e72bdd"}:peer{addr=[2a01:4f9:3a:1055::2]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
The application panicked (crashed).
Message:  called fail_with on already-failed connection state
Location: /Users/rex/rex4539-zebra-peer-set-debug/zebra-network/src/peer/connection.rs:403

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
   0: zebra_network::peer::connection::msg_as_req with msg=inv
      at zebra-network/src/peer/connection.rs:564
   1: zebra_network::peer::handshake::peer with addr=[2a01:4f9:3a:1055::2]:8233
      at zebra-network/src/peer/handshake.rs:198
   2: zebrad::commands:: with zebrad="83e72bdd"
      at zebrad/src/commands.rs:56

zebrad was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 5701 stopped
* thread #12, name = 'tokio-runtime-worker', stop reason = breakpoint 10.6
    frame #0: 0x0000000101705404 zebrad`panic_abort::__rust_start_panic::abort::h19164e12d4c30869 at lib.rs:37:17 [opt]
Target 0: (zebrad) stopped.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #12, name = 'tokio-runtime-worker', stop reason = breakpoint 10.6
  * frame #0: 0x0000000101705404 zebrad`panic_abort::__rust_start_panic::abort::h19164e12d4c30869 at lib.rs:37:17 [opt]
    frame #1: 0x00000001017053e9 zebrad`__rust_start_panic at lib.rs:32:5 [opt]
    frame #2: 0x00000001016f7f89 zebrad`rust_panic at panicking.rs:640:9 [opt]
    frame #3: 0x00000001016f7f3f zebrad`std::panicking::rust_panic_with_hook::hbb70e1d25c7381a9 at panicking.rs:610:5 [opt]
    frame #4: 0x00000001016aafa7 zebrad`std::panicking::begin_panic::_$u7b$$u7b$closure$u7d$$u7d$::hf9c0b57a1c79da14 at panicking.rs:520:9
    frame #5: 0x00000001016abca8 zebrad`std::sys_common::backtrace::__rust_end_short_backtrace::h0e2da18db7995d52(f=<unavailable>) at backtrace.rs:141:18
    frame #6: 0x0000000101728b05 zebrad`std::panicking::begin_panic::h9ef23acf7776d7ab(msg=(data_ptr = "called fail_with on already-failed connection state", length = 51)) at panicking.rs:519:12
    frame #7: 0x00000001001cfbb9 zebrad`zebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::fail_with::ha59aa07ae52c808a(self=0x0000000107881430, e=PeerError @ 0x00007000064430f8) at connection.rs:403:13
    frame #8: 0x00000001001bee9e zebrad`zebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::drive_peer_request::_$u7b$$u7b$closure$u7d$$u7d$::h05c50702d04614ad((null)=ResumeTy @ 0x00007000064444b8) at connection.rs:686:21
    frame #9: 0x00000001001b8a00 zebrad`_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h3ef57d778b32e83a(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x0000700006444880, cx=0x000070000644c1f8) at mod.rs:80:19
    frame #10: 0x00000001001c9f33 zebrad`zebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::handle_message_as_request::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hefc3a654c771665a((null)=ResumeTy @ 0x0000700006447250) at connection.rs:664:9
    frame #11: 0x00000001001bb0e0 zebrad`_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hd8d08677772c8fb2(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x0000700006447a10, cx=0x000070000644c1f8) at mod.rs:80:19
    frame #12: 0x000000010017ce26 zebrad`_$LT$tracing_futures..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h8a6419eff233aa57(self=Pin<&mut tracing_futures::Instrumented<core::future::from_generator::GenFuture<generator-0>>> @ 0x0000700006447a70, cx=0x000070000644c1f8) at lib.rs:258:9
    frame #13: 0x00000001001c5001 zebrad`zebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::handle_message_as_request::_$u7b$$u7b$closure$u7d$$u7d$::h2337c24c8c300ad3((null)=ResumeTy @ 0x0000700006448408) at connection.rs:565:65
    frame #14: 0x00000001001bb6b0 zebrad`_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he532a3acac8ce952(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x0000700006448560, cx=0x000070000644c1f8) at mod.rs:80:19
    frame #15: 0x00000001001ccae6 zebrad`zebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::run::_$u7b$$u7b$closure$u7d$$u7d$::h9f032ec50735c9c3((null)=ResumeTy @ 0x000070000644b960) at connection.rs:325:33
    frame #16: 0x00000001001ba1e0 zebrad`_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h9682f4974cebd320(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x000070000644bff0, cx=0x000070000644c1f8) at mod.rs:80:19
    frame #17: 0x000000010017cdc6 zebrad`_$LT$tracing_futures..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h86941f3385faae37(self=Pin<&mut tracing_futures::Instrumented<core::future::from_generator::GenFuture<generator-0>>> @ 0x000070000644c050, cx=0x000070000644c1f8) at lib.rs:258:9
    frame #18: 0x0000000101146ade zebrad`_$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h622e917129b495f7(self=Pin<&mut core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>> @ 0x000070000644c0b0, cx=0x000070000644c1f8) at future.rs:119:9
    frame #19: 0x00000001010d2b26 zebrad`_$LT$tracing..instrument..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h7a906c52db28be88(self=Pin<&mut tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>>> @ 0x000070000644c130, cx=0x000070000644c1f8) at instrument.rs:151:9
    frame #20: 0x0000000101145d5a zebrad`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h628814ee330d4d0c(ptr=0x0000000105f4ae68) at core.rs:173:17
    frame #21: 0x00000001011106b5 zebrad`tokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::h0676b7d2d2c541da(self=0x0000000105f4ae68, f=closure-0 @ 0x000070000644c278) at unsafe_cell.rs:14:9
    frame #22: 0x0000000101145aa1 zebrad`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h71522dbdd7e32077(self=0x0000000105f4ae60, header=0x0000000105f4ae30) at core.rs:158:13
    frame #23: 0x00000001010c7606 zebrad`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h52bd7bd0b557226e at harness.rs:107:27
    frame #24: 0x00000001010cdaf0 zebrad`core::ops::function::FnOnce::call_once::hfac0dbd1acdb912f((null)=closure-0 @ 0x000070000644c398, (null)=<unavailable>) at function.rs:227:5
    frame #25: 0x000000010114904c zebrad`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hb7f9c0a74e7018d3(self=AssertUnwindSafe<closure-0> @ 0x000070000644c3e0, _args=<unavailable>) at panic.rs:322:9
    frame #26: 0x000000010111b956 zebrad`std::panicking::try::do_call::h5bb2d87997087cc5(data="??D\x06") at panicking.rs:379:40
    frame #27: 0x000000010111b4d9 zebrad`std::panicking::try::h3bdc0c4ac4854659(f=AssertUnwindSafe<closure-0> @ 0x000070000644c5a0) at panicking.rs:343:19
    frame #28: 0x000000010114913c zebrad`std::panic::catch_unwind::h9e29a98a2da32ebe(f=AssertUnwindSafe<closure-0> @ 0x000070000644c610) at panic.rs:396:14
    frame #29: 0x00000001010c6de1 zebrad`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h11caab30e96fb3ad(self=Harness<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> @ 0x000070000644c6b0) at harness.rs:89:19
    frame #30: 0x000000010112f412 zebrad`tokio::runtime::task::raw::poll::h5048e0f4b40e7490(ptr=NonNull<tokio::runtime::task::core::Header> @ 0x000070000644c820) at raw.rs:104:5
    frame #31: 0x000000010160515e zebrad`tokio::runtime::task::raw::RawTask::poll::hc89e5bc150a57c52(self=RawTask @ 0x000070000644c850) at raw.rs:66:18
    frame #32: 0x0000000101646d35 zebrad`tokio::runtime::task::Notified$LT$S$GT$::run::ha2ee5c84333c4156(self=Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> @ 0x000070000644c878) at mod.rs:171:9
    frame #33: 0x0000000101605fec zebrad`tokio::runtime::thread_pool::worker::Context::run_task::_$u7b$$u7b$closure$u7d$$u7d$::hab67772daa1796dd at worker.rs:346:13
    frame #34: 0x0000000101643fde zebrad`tokio::coop::with_budget::_$u7b$$u7b$closure$u7d$$u7d$::h8844a16f5ff90b15(cell=0x0000000109008e22) at coop.rs:121:9
    frame #35: 0x000000010165ce3b zebrad`std::thread::local::LocalKey$LT$T$GT$::try_with::hbd5fcf16e5f1c295(self=0x0000000101996e48, f=closure-0 @ 0x000070000644ca90) at local.rs:272:16
    frame #36: 0x000000010165bd1c zebrad`std::thread::local::LocalKey$LT$T$GT$::with::hda68ae8d7d88b19a(self=0x0000000101996e48, f=<unavailable>) at local.rs:248:9
    frame #37: 0x0000000101605f93 zebrad`tokio::runtime::thread_pool::worker::Context::run_task::he75bd550cde8a541 [inlined] tokio::coop::with_budget::h1542830cccad9c0d(budget=(__0 = core::option::Option<u8> @ 0x000070000644cb68), f=closure-0 @ 0x000070000644cb70) at coop.rs:114:5
    frame #38: 0x0000000101605f77 zebrad`tokio::runtime::thread_pool::worker::Context::run_task::he75bd550cde8a541 [inlined] tokio::coop::budget::hbd3a5760b4b06b29(f=closure-0 @ 0x000070000644cb40) at coop.rs:98
    frame #39: 0x0000000101605f47 zebrad`tokio::runtime::thread_pool::worker::Context::run_task::he75bd550cde8a541(self=0x000070000644cd78, task=Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> @ 0x000070000644cb30, core=0x0000000105908fa0) at worker.rs:345
    frame #40: 0x0000000101605cb7 zebrad`tokio::runtime::thread_pool::worker::Context::run::hb0246eebe05e624c(self=0x000070000644cd78, core=0x0000000105908fa0) at worker.rs:317:24
    frame #41: 0x0000000101605b95 zebrad`tokio::runtime::thread_pool::worker::run::_$u7b$$u7b$closure$u7d$$u7d$::h8e5b550afab8865d at worker.rs:302:17
    frame #42: 0x0000000101654ef3 zebrad`tokio::macros::scoped_tls::ScopedKey$LT$T$GT$::set::hc7b1486b701f362f(self=0x0000000101997108, t=0x000070000644cd78, f=closure-0 @ 0x000070000644cd08) at scoped_tls.rs:61:9
    frame #43: 0x0000000101605b58 zebrad`tokio::runtime::thread_pool::worker::run::h8f2f8de5d0b46d36(worker=Arc<tokio::runtime::thread_pool::worker::Worker> @ 0x000070000644cd60) at worker.rs:299:5
    frame #44: 0x0000000101605a61 zebrad`tokio::runtime::thread_pool::worker::Launch::launch::_$u7b$$u7b$closure$u7d$$u7d$::h140892baace04f3e at worker.rs:278:45
    frame #45: 0x0000000101620538 zebrad`tokio::runtime::handle::Handle::spawn_blocking::_$u7b$$u7b$closure$u7d$$u7d$::hf5b5485d1b6e6d8d at handle.rs:196:17
    frame #46: 0x000000010161a3ad zebrad`_$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h644b2a0df9510d47(self=Pin<&mut tokio::runtime::blocking::task::BlockingTask<closure-0>> @ 0x000070000644ce40, _cx=0x000070000644cf88) at task.rs:42:21
    frame #47: 0x000000010161f27a zebrad`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::hd12bee239e809f69(ptr=0x0000000105f04650) at core.rs:173:17
    frame #48: 0x000000010162e795 zebrad`tokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::he8150f8a6a43b4fd(self=0x0000000105f04650, f=closure-0 @ 0x000070000644d008) at unsafe_cell.rs:14:9
    frame #49: 0x000000010161f11b zebrad`tokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::hd9228082ccf0d0e9(self=0x0000000105f04650, header=0x0000000105f04620) at core.rs:158:13
    frame #50: 0x0000000101648dc6 zebrad`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::hb9c59f2cbebf8957 at harness.rs:107:27
    frame #51: 0x00000001016615c0 zebrad`core::ops::function::FnOnce::call_once::hc1451c26209a52cb((null)=closure-0 @ 0x000070000644d128, (null)=<unavailable>) at function.rs:227:5
    frame #52: 0x000000010162119c zebrad`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h4d7f8d3bbf2d6c76(self=AssertUnwindSafe<closure-0> @ 0x000070000644d170, _args=<unavailable>) at panic.rs:322:9
    frame #53: 0x0000000101654bb6 zebrad`std::panicking::try::do_call::hf3bd26e5cd7d5ccb(data="@?D\x06") at panicking.rs:379:40
    frame #54: 0x00000001016547c9 zebrad`std::panicking::try::h6744a753065b3609(f=AssertUnwindSafe<closure-0> @ 0x000070000644d330) at panicking.rs:343:19
    frame #55: 0x000000010162218c zebrad`std::panic::catch_unwind::h0e6df765d0ec468f(f=AssertUnwindSafe<closure-0> @ 0x000070000644d3a0) at panic.rs:396:14
    frame #56: 0x0000000101648a91 zebrad`tokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h3fdafeeae85a3c02(self=Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule> @ 0x000070000644d440) at harness.rs:89:19
    frame #57: 0x00000001016052a2 zebrad`tokio::runtime::task::raw::poll::hef8c96fa66648c7c(ptr=NonNull<tokio::runtime::task::core::Header> @ 0x000070000644d5b0) at raw.rs:104:5
    frame #58: 0x000000010160515e zebrad`tokio::runtime::task::raw::RawTask::poll::hc89e5bc150a57c52(self=RawTask @ 0x000070000644d5e0) at raw.rs:66:18
    frame #59: 0x0000000101646d05 zebrad`tokio::runtime::task::Notified$LT$S$GT$::run::h3b1927b24bcba4fa(self=Notified<tokio::runtime::blocking::schedule::NoopSchedule> @ 0x000070000644d608) at mod.rs:171:9
    frame #60: 0x0000000101612546 zebrad`tokio::runtime::blocking::pool::Inner::run::h1dbc71d4a7461847(self=0x0000000105909650, worker_thread_id=10) at pool.rs:277:17
    frame #61: 0x00000001016123f5 zebrad`tokio::runtime::blocking::pool::Spawner::spawn_thread::_$u7b$$u7b$closure$u7d$$u7d$::hcf03f17bdb61a19e at pool.rs:257:17
    frame #62: 0x00000001015fb384 zebrad`std::sys_common::backtrace::__rust_begin_short_backtrace::h4f8b7f40a5d5b31a(f=<unavailable>) at backtrace.rs:125:18
    frame #63: 0x00000001016236e1 zebrad`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hff51272d10ede402 at mod.rs:474:17
    frame #64: 0x00000001016211e1 zebrad`_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h6f19d153361e8a6d(self=<unavailable>, _args=<unavailable>) at panic.rs:322:9
    frame #65: 0x0000000101654b25 zebrad`std::panicking::try::do_call::h2fb91f3a2381eb34(data="\x01") at panicking.rs:379:40
    frame #66: 0x0000000101654a33 zebrad`std::panicking::try::h99adac12e42dfae6(f=<unavailable>) at panicking.rs:343:19
    frame #67: 0x0000000101622201 zebrad`std::panic::catch_unwind::he3386189a2f52b0d(f=<unavailable>) at panic.rs:396:14
    frame #68: 0x00000001016235fd zebrad`std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::hdaf74289515ee884 at mod.rs:473:30
    frame #69: 0x0000000101661151 zebrad`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hb5cfd68db5d3e283((null)=0x000000010590ac10, (null)=<unavailable>) at function.rs:227:5
    frame #70: 0x00000001016fb76d zebrad`std::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h1658d06ba2d41cb5 at boxed.rs:1328:9 [opt]
    frame #71: 0x00000001016fb767 zebrad`std::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h747cc083e7349205 at boxed.rs:1328 [opt]
    frame #72: 0x00000001016fb75e zebrad`std::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 at thread.rs:71 [opt]
    frame #73: 0x00007fff20357950 libsystem_pthread.dylib`_pthread_start + 224
    frame #74: 0x00007fff2035347b libsystem_pthread.dylib`thread_start + 15

Error

called fail_with on already-failed connection state

Metadata

key value
version 3.0.0-alpha.0
git commit 83e72bdd
location /Users/rex/rex4539-zebra-peer-set-debug/zebra-network/src/peer/connection.rs:403:13

SpanTrace

SpanTrace:
   0: zebra_network::peer::connection::msg_as_req
           with msg=inv
             at zebra-network/src/peer/connection.rs:564
   1: zebra_network::peer::handshake::peer
           with addr=[2a01:4f9:3a:1055::2]:8233
             at zebra-network/src/peer/handshake.rs:198
   2: zebrad::commands::
           with zebrad="83e72bdd"
             at zebrad/src/commands.rs:56

BackTrace

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: backtrace::backtrace::libunwind::trace::h56b5b89d46ea18ee
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/libunwind.rs:90
        88 │ #[inline(always)]
        89 │ pub unsafe fn trace(mut cb: &mut dyn FnMut(&super::Frame) -> bool) {
        90 >     uw::_Unwind_Backtrace(trace_fn, &mut cb as *mut _ as *mut _);
        91 │ 
        92 │     extern "C" fn trace_fn(
   2: backtrace::backtrace::trace_unsynchronized::h456d388ebbb97b58
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:66
        64 │ /// See information on `trace` for caveats on `cb` panicking.
        65 │ pub unsafe fn trace_unsynchronized<F: FnMut(&Frame) -> bool>(mut cb: F) {
        66 >     trace_imp(&mut cb)
        67 │ }
        68 │ 
   3: backtrace::backtrace::trace::h0c2ee1e6653540fd
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:53
        51 │ pub fn trace<F: FnMut(&Frame) -> bool>(cb: F) {
        52 │     let _guard = crate::lock::lock();
        53 >     unsafe { trace_unsynchronized(cb) }
        54 │ }
        55 │ 
   4: backtrace::capture::Backtrace::create::h8d054d598eb8a93e
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:176
       174 │         let mut frames = Vec::new();
       175 │         let mut actual_start_index = None;
       176 >         trace(|frame| {
       177 │             frames.push(BacktraceFrame {
       178 │                 frame: Frame::Raw(frame.clone()),
   5: backtrace::capture::Backtrace::new::h89e80605e5dd4af5
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:140
       138 │     #[inline(never)] // want to make sure there's a frame here to remove
       139 │     pub fn new() -> Backtrace {
       140 >         let mut bt = Self::create(Self::new as usize);
       141 │         bt.resolve();
       142 │         bt
   6: color_eyre::config::PanicHook::panic_report::h35620cffb3b3e3a3
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:982
       980 │ 
       981 │         let backtrace = if capture_bt {
       982 >             Some(backtrace::Backtrace::new())
       983 │         } else {
       984 │             None
@rex4539
Copy link
Contributor Author

rex4539 commented Dec 13, 2020

I keep getting this even after trashing all database.

Click triangle to reveal logs

Dec 13 12:59:10.511  INFO {zebrad="e537926e"}: zebrad::commands::start: Starting zebrad
Dec 13 12:59:10.512  INFO {zebrad="e537926e"}: zebrad::commands::start: config=ZebradConfig { consensus: Config { checkpoint_sync: false }, metrics: MetricsSection { endpoint_addr: None }, network: Config { listen_addr: 0.0.0.0:8233, network: Mainnet, initial_mainnet_peers: {"mainnet.is.yolo.money:8233", "dnsseed.str4d.xyz:8233", "mainnet.seeder.zfnd.org:8233", "dnsseed.z.cash:8233"}, initial_testnet_peers: {"testnet.is.yolo.money:18233", "testnet.seeder.zfnd.org:18233", "dnsseed.testnet.z.cash:18233"}, peerset_initial_target_size: 50, new_peer_interval: 60s }, state: Config { cache_dir: "/Users/rex/Library/Caches/zebra", ephemeral: false, debug_stop_at_height: None }, tracing: TracingSection { use_color: true, filter: None, endpoint_addr: None, flamegraph: None }, sync: SyncSection { max_concurrent_block_requests: 50, lookahead_limit: 2000 } }
Dec 13 12:59:10.513  INFO {zebrad="e537926e"}: zebrad::commands::start: initializing node state
Dec 13 12:59:10.515  INFO {zebrad="e537926e"}: zebra_state::config: set the open file limit for Zebra new_limit=1024 old_limit=Some(256) hard_rlimit=Some(9223372036854775807)
Dec 13 12:59:10.575  INFO {zebrad="e537926e"}: zebrad::commands::start: initializing chain verifier
Dec 13 12:59:10.605  INFO {zebrad="e537926e"}:init{config=Config { checkpoint_sync: false } network=Mainnet}: zebra_consensus::chain: initializing chain verifier tip=None max_checkpoint_height=Height(419581)
Dec 13 12:59:10.605  INFO {zebrad="e537926e"}: zebrad::commands::start: initializing network
Dec 13 12:59:10.822  INFO {zebrad="e537926e"}: zebra_network::peer_set::initialize: Sending initial request for peers
Dec 13 12:59:10.822  INFO add_initial_peers: zebra_network::peer_set::initialize: Connecting to initial peer set initial_peers={18.139.208.246:8233, 61.160.196.169:8233, 116.202.13.16:8233, 104.248.144.72:8233, 147.135.11.144:8233, 47.254.179.213:8233, 159.65.12.52:8233, 165.22.76.97:8233, 95.217.78.170:8233, [2a01:4f8:120:72a3::2]:8233, 47.254.130.139:8233, 204.228.147.42:8233, 195.201.110.220:8233, 176.9.78.6:8233, 195.200.244.73:8233, 47.252.79.236:8233, 161.97.68.118:8233, 195.201.11.122:8233, [2607:5300:203:f62::]:8233, 95.179.179.21:8233, 172.104.106.153:8233, 50.220.121.211:8233, 167.172.185.201:8233, 157.245.172.190:8233, 116.202.129.35:8233, 3.112.252.179:8233, [2a01:4f9:c010:7391::1]:8233, 47.90.200.114:8233, 67.23.181.122:8233, 199.247.26.15:8233, 116.202.132.28:8233, 139.162.138.168:8233, 3.139.233.192:8233, 195.201.11.54:8233, 47.93.194.27:8233, 39.97.255.168:8233, 34.87.37.112:8233, 149.56.135.165:8233, [2a01:4f8:13b:cc5::2]:8233, 206.189.79.10:8233, 34.67.71.42:8233, 161.53.201.181:8233, 31.31.73.46:8233, 159.65.179.43:8233, 161.53.201.188:8233, 91.206.16.214:8233, [2a02:c207:2022:9236::1]:8233, 94.156.174.100:8233, [2a01:4f9:4a:1e15::2]:8233, 50.213.167.113:8233, [2a01:4f8:b0:920a::5502]:8233, 95.217.155.210:8233, [2a02:7b40:c287:513d::1]:8233, [2a01:4f8:151:120d::2]:8233, 80.242.124.21:8233, 173.212.197.63:8233, 39.97.242.143:8233, 47.100.175.51:8233, 144.217.253.98:8233, 93.51.53.177:8233, 47.252.87.61:8233, 173.249.20.4:8233, 135.181.49.198:8233, 47.75.53.157:8233, 47.252.26.121:8233, 138.201.120.49:8233, [2a01:4f9:4b:42cb::2]:8233, 138.201.252.11:8233, 46.28.111.180:8233, 82.64.95.124:8233, 67.183.29.123:8233, 134.209.156.32:8233, 51.75.146.76:8233, 46.4.65.10:8233, 135.181.136.106:8233, 51.89.96.117:8233, [2a01:4f8:140:2393::2]:8233, 95.179.182.168:8233, 35.238.100.203:8233, 74.207.236.134:8233, 139.196.121.177:8233, 51.77.64.51:8233}
Dec 13 12:59:11.410  INFO {zebrad="e537926e"}: zebrad::commands::start: initializing syncer
Dec 13 12:59:11.411  INFO {zebrad="e537926e"}:sync: zebrad::components::sync: starting genesis block download and verify
Dec 13 12:59:11.413  WARN {zebrad="e537926e"}:sync: zebrad::components::sync: could not download or verify genesis block, retrying e=ServiceError { inner: Os { code: 48, kind: AddrInUse, message: "Address already in use" } }
Dec 13 12:59:11.413  INFO {zebrad="e537926e"}:sync: zebrad::components::sync: starting genesis block download and verify
Dec 13 12:59:11.431 ERROR {zebrad="e537926e"}:peer{addr=39.97.255.168:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Dec 13 12:59:11.510  WARN {zebrad="e537926e"}:peer{addr=34.87.37.112:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Dec 13 12:59:12.145  WARN {zebrad="e537926e"}:peer{addr=39.97.242.143:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Dec 13 12:59:13.441  WARN {zebrad="e537926e"}:peer{addr=[2607:5300:203:f62::]:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Dec 13 12:59:13.445  WARN {zebrad="e537926e"}:peer{addr=47.252.26.121:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Dec 13 12:59:14.198  WARN {zebrad="e537926e"}:peer{addr=206.189.79.10:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Dec 13 12:59:15.181  WARN {zebrad="e537926e"}:peer{addr=47.252.79.236:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
The application panicked (crashed).
Message:  called fail_with on already-failed connection state
Location: /Users/rex/zebra/zebra-network/src/peer/connection.rs:403

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
   0: zebra_network::peer::connection::msg_as_req with msg=getheaders
      at zebra-network/src/peer/connection.rs:564
   1: zebra_network::peer::handshake::peer with addr=34.87.37.112:8233
      at zebra-network/src/peer/handshake.rs:198
   2: zebrad::commands:: with zebrad="e537926e"
      at zebrad/src/commands.rs:56

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: backtrace::backtrace::libunwind::trace::h56b5b89d46ea18ee
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/libunwind.rs:90
        88 │ #[inline(always)]
        89 │ pub unsafe fn trace(mut cb: &mut dyn FnMut(&super::Frame) -> bool) {
        90 >     uw::_Unwind_Backtrace(trace_fn, &mut cb as *mut _ as *mut _);
        91 │ 
        92 │     extern "C" fn trace_fn(
   2: backtrace::backtrace::trace_unsynchronized::h456d388ebbb97b58
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:66
        64 │ /// See information on `trace` for caveats on `cb` panicking.
        65 │ pub unsafe fn trace_unsynchronized<F: FnMut(&Frame) -> bool>(mut cb: F) {
        66 >     trace_imp(&mut cb)
        67 │ }
        68 │ 
   3: backtrace::backtrace::trace::h0c2ee1e6653540fd
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:53
        51 │ pub fn trace<F: FnMut(&Frame) -> bool>(cb: F) {
        52 │     let _guard = crate::lock::lock();
        53 >     unsafe { trace_unsynchronized(cb) }
        54 │ }
        55 │ 
   4: backtrace::capture::Backtrace::create::h8d054d598eb8a93e
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:176
       174 │         let mut frames = Vec::new();
       175 │         let mut actual_start_index = None;
       176 >         trace(|frame| {
       177 │             frames.push(BacktraceFrame {
       178 │                 frame: Frame::Raw(frame.clone()),
   5: backtrace::capture::Backtrace::new::h89e80605e5dd4af5
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:140
       138 │     #[inline(never)] // want to make sure there's a frame here to remove
       139 │     pub fn new() -> Backtrace {
       140 >         let mut bt = Self::create(Self::new as usize);
       141 │         bt.resolve();
       142 │         bt
   6: color_eyre::config::PanicHook::panic_report::h35620cffb3b3e3a3
      at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:982
       980 │ 
       981 │         let backtrace = if capture_bt {
       982 >             Some(backtrace::Backtrace::new())
       983 │         } else {
       984 │             None

@rex4539
Copy link
Contributor Author

rex4539 commented Dec 13, 2020

There was a stale zebrad process, which caused all the subsequent crashes. After I killed it, all is good.

@oxarbitrage
Copy link
Contributor

I am not totally sure but i think that if we do a port already in use check before starting any networking process as suggested at #1484 (comment) we can avoid this issue.

@teor2345
Copy link
Contributor

I am not totally sure but i think that if we do a port already in use check before starting any networking process as suggested at #1484 (comment) we can avoid this issue.

This panic happens due to a bug in zebra-network's error handling. zebra-network should not be calling fail_with on already-failed connection states - it doesn't matter how the connection originally failed.

Fixing the misuse of fail_with will fix this bug, regardless of the initial cause. Detecting port conflicts earlier will only avoid one possible source of connection failures.

@teor2345
Copy link
Contributor

We should also check the other assumptions made by the inbound and sync services. We know there's at least one other bug where the network code handles connection state incorrectly: #1435 - peer connection permanent failure.

@rex4539
Copy link
Contributor Author

rex4539 commented Jan 3, 2021

Got this again.

Logs

Click triangle to reveal details

Jan 03 23:37:56.347  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: Starting zebrad
Jan 03 23:37:56.348  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: config=ZebradConfig { consensus: Config { checkpoint_sync: false }, metrics: MetricsSection { endpoint_addr: None }, network: Config { listen_addr: 0.0.0.0:8233, network: Mainnet, initial_mainnet_peers: {"dnsseed.str4d.xyz:8233", "mainnet.is.yolo.money:8233", "mainnet.seeder.zfnd.org:8233", "dnsseed.z.cash:8233"}, initial_testnet_peers: {"testnet.seeder.zfnd.org:18233", "dnsseed.testnet.z.cash:18233", "testnet.is.yolo.money:18233"}, peerset_initial_target_size: 50, new_peer_interval: 60s }, state: Config { cache_dir: "/Users/rex/Library/Caches/zebra", ephemeral: false, debug_stop_at_height: None }, tracing: TracingSection { use_color: true, filter: None, endpoint_addr: None, flamegraph: None }, sync: SyncSection { max_concurrent_block_requests: 50, lookahead_limit: 2000 } }
Jan 03 23:37:56.348  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: initializing node state
Jan 03 23:37:56.348  INFO {zebrad="f9eb4a28"}: zebra_state::config: the open file limit is at or above the specified limit new_limit=1024 current_limit=2560 hard_rlimit=Some(9223372036854775807)
Jan 03 23:37:57.499  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: initializing chain verifier
Jan 03 23:37:57.520  INFO {zebrad="f9eb4a28"}:init{config=Config { checkpoint_sync: false } network=Mainnet}: zebra_consensus::chain: initializing chain verifier tip=Some((Height(1095386), block::Hash("0000000001d1b6e39156eaa6edcd18fb1aca2864133b39716ff29bffb6c64c87"))) max_checkpoint_height=Height(419581)
Jan 03 23:37:57.520  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: initializing network
Jan 03 23:37:59.024  INFO {zebrad="f9eb4a28"}: zebra_network::peer_set::initialize: Sending initial request for peers
Jan 03 23:37:59.024  INFO add_initial_peers: zebra_network::peer_set::initialize: Connecting to initial peer set initial_peers={159.65.179.43:8233, 135.181.136.106:8233, 46.4.65.10:8233, 172.104.106.153:8233, 61.160.196.169:8233, 37.157.195.61:8233, 47.254.179.213:8233, 47.252.80.131:8233, [2a01:4f8:141:40d8::2]:8233, [2001:41d0:8:cdd7::]:8233, 47.75.53.157:8233, 93.51.53.177:8233, 47.252.79.236:8233, 138.201.120.49:8233, 51.89.96.118:8233, 52.52.174.26:8233, 85.15.179.171:8233, 51.77.64.61:8233, 209.141.47.197:8233, 50.220.121.211:8233, 158.69.252.212:8233, 91.206.16.214:8233, 51.77.64.51:8233, 135.181.49.198:8233, 34.90.231.182:8233, 47.94.92.186:8233, 173.249.20.4:8233, 5.135.140.215:8233, [2001:41d0:700:223d::]:8233, 74.207.236.134:8233, [2a02:7b40:c287:513d::1]:8233, 51.75.146.76:8233, 116.202.132.28:8233, 151.80.96.105:8233, 95.179.179.21:8233, 116.202.13.16:8233, 109.167.229.52:8233, 34.66.191.95:8233, [2001:41d0:303:d64b::]:8233, 46.165.230.156:8233, 104.131.113.161:8233, 3.92.60.219:8233, 59.110.243.62:8233, 51.77.64.59:8233, 143.110.216.2:8233, 39.97.242.143:8233, 95.216.46.146:8233, 138.197.174.176:8233, 149.56.135.165:8233, 50.213.167.113:8233, [2001:41d0:700:2233::]:8233, 54.38.136.58:8233, [2a01:4f8:13b:cc5::2]:8233, 139.162.138.168:8233, 13.231.145.68:8233, 35.198.119.137:8233, 116.202.170.226:8233, 39.97.246.221:8233, 47.100.175.51:8233, 47.107.242.15:8233, [2a01:4f8:b0:920a::5502]:8233, 198.187.30.208:8233, 159.65.12.52:8233, 116.203.188.195:8233, 195.201.11.54:8233, 213.239.201.46:8233, 47.253.0.14:8233, 152.54.8.147:8233, 47.254.69.198:8233, 47.93.194.27:8233, 78.46.43.59:8233, [2001:41d0:d:27d1:1234:5678:90ab:cdef]:8233, 31.31.73.46:8233, [2001:41d0:8:ce33::]:8233, 161.97.68.118:8233, 161.35.171.22:8233, 195.201.60.233:8233, [2a01:4f8:140:2393::2]:8233, 67.23.181.122:8233, 95.179.182.168:8233, 172.104.180.119:8233, [2a01:4f9:4a:1e15::2]:8233, 77.93.223.9:8233}
Jan 03 23:37:59.279  WARN {zebrad="f9eb4a28"}:peer{addr=158.69.252.212:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.295  WARN {zebrad="f9eb4a28"}:peer{addr=34.66.191.95:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.299  WARN {zebrad="f9eb4a28"}:peer{addr=104.131.113.161:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.359  WARN {zebrad="f9eb4a28"}:peer{addr=52.52.174.26:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.373  WARN {zebrad="f9eb4a28"}:peer{addr=209.141.47.197:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.373  WARN {zebrad="f9eb4a28"}:peer{addr=198.187.30.208:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.418  WARN {zebrad="f9eb4a28"}:peer{addr=59.110.243.62:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.429  WARN {zebrad="f9eb4a28"}:peer{addr=195.201.60.233:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.482  INFO {zebrad="f9eb4a28"}:peer{addr=[2a02:7b40:c287:513d::1]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:37:59.491  WARN {zebrad="f9eb4a28"}:peer{addr=39.97.246.221:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.511  WARN {zebrad="f9eb4a28"}:peer{addr=47.107.242.15:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.519  WARN {zebrad="f9eb4a28"}:peer{addr=47.100.175.51:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.555  WARN {zebrad="f9eb4a28"}:peer{addr=13.231.145.68:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.592  WARN {zebrad="f9eb4a28"}:peer{addr=47.75.53.157:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.601  WARN {zebrad="f9eb4a28"}:peer{addr=39.97.242.143:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:37:59.604  WARN {zebrad="f9eb4a28"}:peer{addr=47.93.194.27:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:00.121  INFO {zebrad="f9eb4a28"}:peer{addr=95.216.46.146:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:00.748  INFO {zebrad="f9eb4a28"}:peer{addr=135.181.136.106:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:01.464  WARN {zebrad="f9eb4a28"}:peer{addr=47.254.69.198:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:01.471  INFO {zebrad="f9eb4a28"}:peer{addr=[2a01:4f9:4a:1e15::2]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:02.041  INFO {zebrad="f9eb4a28"}: zebrad::commands::start: initializing syncer
Jan 03 23:38:02.157  INFO {zebrad="f9eb4a28"}:peer{addr=95.179.182.168:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:02.827  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:700:223d::]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=51.89.96.118:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=[2a01:4f8:13b:cc5::2]:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=93.51.53.177:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=51.77.64.59:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=51.77.64.51:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=54.38.136.58:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=47.252.79.236:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=47.252.80.131:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=47.94.92.186:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=149.56.135.165:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=74.207.236.134:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=47.253.0.14:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=152.54.8.147:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=47.254.179.213:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=143.110.216.2:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=193.42.110.30:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.125 ERROR {zebrad="f9eb4a28"}:peer{addr=3.92.60.219:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.126 ERROR {zebrad="f9eb4a28"}:peer{addr=138.197.174.176:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.126 ERROR {zebrad="f9eb4a28"}:peer{addr=85.15.179.171:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.126 ERROR {zebrad="f9eb4a28"}:peer{addr=91.206.16.214:8233}:msg_as_req{msg=getheaders}: zebra_network::peer::connection: e=buffered service failed: buffered service failed: Address already in use (os error 48)
Jan 03 23:38:03.510  INFO {zebrad="f9eb4a28"}:peer{addr=95.179.179.21:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:04.131  INFO {zebrad="f9eb4a28"}:peer{addr=46.165.230.156:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:04.772  INFO {zebrad="f9eb4a28"}:peer{addr=34.90.231.182:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:05.394  INFO {zebrad="f9eb4a28"}:peer{addr=116.203.188.195:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:06.211  INFO {zebrad="f9eb4a28"}:peer{addr=161.35.171.22:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:06.682  WARN {zebrad="f9eb4a28"}:peer{addr=116.203.188.195:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:06.874  INFO {zebrad="f9eb4a28"}:peer{addr=35.198.119.137:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:07.316  WARN {zebrad="f9eb4a28"}:peer{addr=161.35.171.22:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:07.515  INFO {zebrad="f9eb4a28"}:peer{addr=78.46.43.59:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:08.163  INFO {zebrad="f9eb4a28"}:peer{addr=116.202.170.226:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:08.673  WARN {zebrad="f9eb4a28"}:peer{addr=78.46.43.59:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:08.861  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:8:cdd7::]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:09.317  WARN {zebrad="f9eb4a28"}:peer{addr=116.202.170.226:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:09.675  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:303:d64b::]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:10.177  WARN {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:8:cdd7::]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:10.315  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:8:ce33::]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:10.777  WARN {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:303:d64b::]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:10.948  INFO {zebrad="f9eb4a28"}:peer{addr=[2a01:4f8:141:40d8::2]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:11.451  WARN {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:8:ce33::]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:11.590  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:d:27d1:1234:5678:90ab:cdef]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:12.061  WARN {zebrad="f9eb4a28"}:peer{addr=[2a01:4f8:141:40d8::2]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:12.234  INFO {zebrad="f9eb4a28"}:peer{addr=213.239.201.46:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:12.690  WARN {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:d:27d1:1234:5678:90ab:cdef]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:12.866  INFO {zebrad="f9eb4a28"}:peer{addr=51.75.146.76:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:13.493  INFO {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:700:2233::]:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:13.956  WARN {zebrad="f9eb4a28"}:peer{addr=51.75.146.76:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:14.125  INFO {zebrad="f9eb4a28"}:peer{addr=5.135.140.215:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:14.617  WARN {zebrad="f9eb4a28"}:peer{addr=[2001:41d0:700:2233::]:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
Jan 03 23:38:14.780  INFO {zebrad="f9eb4a28"}:peer{addr=51.77.64.61:8233}:msg_as_req{msg=getheaders}:inbound:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1091479) response_len=158 chain_tip_height=Height(1095386) stop_height=None intersection_height=Some(Height(1091321))
Jan 03 23:38:15.077  INFO {zebrad="f9eb4a28"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Jan 03 23:38:15.130  INFO {zebrad="f9eb4a28"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(1095386) min_locator_height=1095287 locators=[Height(1095386), Height(1095385), Height(1095384), Height(1095382), Height(1095378), Height(1095370), Height(1095354), Height(1095322), Height(1095287)]

Panic

Jan 03 23:38:15.310  WARN {zebrad="f9eb4a28"}:peer{addr=5.135.140.215:8233}:msg_as_req{msg=inv}: zebra_network::peer::connection: inbound service is overloaded, closing connection
The application panicked (crashed).
Message:  called fail_with on already-failed connection state
Location: /Users/rex/zebra/zebra-network/src/peer/connection.rs:403

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  
   0: zebra_network::peer::connection::msg_as_req with msg=inv
      at zebra-network/src/peer/connection.rs:564
   1: zebra_network::peer::handshake::peer with addr=116.202.170.226:8233
      at zebra-network/src/peer/handshake.rs:198
   2: zebrad::commands:: with zebrad="f9eb4a28"
      at zebrad/src/commands.rs:56

Detailed Backtrace

``` ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1: backtrace::backtrace::libunwind::trace::h56b5b89d46ea18ee at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/libunwind.rs:90 88 │ #[inline(always)] 89 │ pub unsafe fn trace(mut cb: &mut dyn FnMut(&super::Frame) -> bool) { 90 > uw::_Unwind_Backtrace(trace_fn, &mut cb as *mut _ as *mut _); 91 │ 92 │ extern "C" fn trace_fn( 2: backtrace::backtrace::trace_unsynchronized::h35ba3908cf2057d6 at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:66 64 │ /// See information on `trace` for caveats on `cb` panicking. 65 │ pub unsafe fn trace_unsynchronized bool>(mut cb: F) { 66 > trace_imp(&mut cb) 67 │ } 68 │ 3: backtrace::backtrace::trace::h0b6131d78bbf6f4f at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/backtrace/mod.rs:53 51 │ pub fn trace bool>(cb: F) { 52 │ let _guard = crate::lock::lock(); 53 > unsafe { trace_unsynchronized(cb) } 54 │ } 55 │ 4: backtrace::capture::Backtrace::create::h8d054d598eb8a93e at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:176 174 │ let mut frames = Vec::new(); 175 │ let mut actual_start_index = None; 176 > trace(|frame| { 177 │ frames.push(BacktraceFrame { 178 │ frame: Frame::Raw(frame.clone()), 5: backtrace::capture::Backtrace::new::h89e80605e5dd4af5 at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.54/src/capture.rs:140 138 │ #[inline(never)] // want to make sure there's a frame here to remove 139 │ pub fn new() -> Backtrace { 140 > let mut bt = Self::create(Self::new as usize); 141 │ bt.resolve(); 142 │ bt 6: color_eyre::config::PanicHook::panic_report::ha2f85b9c0f7cb762 at /Users/rex/.cargo/registry/src/github.com-1ecc6299db9ec823/color-eyre-0.5.10/src/config.rs:982 980 │ 981 │ let backtrace = if capture_bt { 982 > Some(backtrace::Backtrace::new()) 983 │ } else { 984 │ None

The application panicked (crashed).
Message: called fail_with on already-failed connection state
Location: /Users/rex/zebra/zebra-network/src/peer/connection.rs:403zebrad was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 9882 stopped

  • thread Start of tower-based networking. #5, name = 'tokio-runtime-worker', stop reason = breakpoint 10.6
    frame #0: 0x0000000101710fc4 zebrad`panic_abort::__rust_start_panic::abort::h19164e12d4c30869 at lib.rs:39:17 [opt]
    Target 0: (zebrad) stopped.
    (lldb) bt
    error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
    error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
    error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
    error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
    error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
  • thread Start of tower-based networking. #5, name = 'tokio-runtime-worker', stop reason = breakpoint 10.6
    • frame #0: 0x0000000101710fc4 zebradpanic_abort::__rust_start_panic::abort::h19164e12d4c30869 at lib.rs:39:17 [opt] frame #1: 0x0000000101710fa9 zebrad__rust_start_panic at lib.rs:34:5 [opt]
      frame Continuous integration #2: 0x0000000101708559 zebradrust_panic at panicking.rs:640:9 [opt] frame #3: 0x000000010170850f zebradstd::panicking::rust_panic_with_hook::h102f8bee4e0ef4c7 at panicking.rs:610:5 [opt]
      frame Add a skeleton enum for network messages. #4: 0x00000001016b62b7 zebradstd::panicking::begin_panic::_$u7b$$u7b$closure$u7d$$u7d$::hf9c0b57a1c79da14 at panicking.rs:520:9 frame #5: 0x00000001016b6fb8 zebradstd::sys_common::backtrace::__rust_end_short_backtrace::h0e2da18db7995d52(f=) at backtrace.rs:141:18
      frame Move zebra-chain::types::Transaction to its own module. #6: 0x0000000101734955 zebradstd::panicking::begin_panic::h9ef23acf7776d7ab(msg="called fail_with on already-failed connection state") at panicking.rs:519:12 frame #7: 0x0000000100216529 zebradzebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::fail_with::h0d0690a47b2ffac0(self=0x0000000109814230, e=PeerError @ 0x000070000e6d00f8) at connection.rs:403:13
      frame Checksum(Read|Writ)er #8: 0x000000010020580e zebradzebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::drive_peer_request::_$u7b$$u7b$closure$u7d$$u7d$::ha35f1c7f8bf22a39((null)=ResumeTy @ 0x000070000e6d14b8) at connection.rs:686:21 frame #9: 0x00000001003567f0 zebrad$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::he05e4f84d41461e9(self=Pin<&mut core::future::from_generator::GenFuture> @ 0x000070000e6d1880, cx=0x000070000e6d91f8) at mod.rs:80:19
      frame Add a helper method for reading fixed-size byte arrays #10: 0x00000001002108a3 zebradzebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::handle_message_as_request::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hd8ee14ec683350d2((null)=ResumeTy @ 0x000070000e6d4250) at connection.rs:664:9 frame #11: 0x0000000100354670 zebrad
      $LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h66647bb670ca37fe(self=Pin<&mut core::future::from_generator::GenFuture> @ 0x000070000e6d4a10, cx=0x000070000e6d91f8) at mod.rs:80:19
      frame Replace Version MetaAddr with (Services, SocketAddr). #12: 0x0000000100273db6 zebrad_$LT$tracing_futures..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h2972b52d5cc1e777(self=Pin<&mut tracing_futures::Instrumented<core::future::from_generator::GenFuture<generator-0>>> @ 0x000070000e6d4a70, cx=0x000070000e6d91f8) at lib.rs:258:9 frame #13: 0x000000010020b971 zebradzebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::handle_message_as_request::$u7b$$u7b$closure$u7d$$u7d$::hbc684148e471ff73((null)=ResumeTy @ 0x000070000e6d5408) at connection.rs:565:65
      frame Clean tracing_subscriber deprecation warnings. #14: 0x0000000100355cc0 zebrad_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::had82af07c3926bf8(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x000070000e6d5560, cx=0x000070000e6d91f8) at mod.rs:80:19 frame #15: 0x0000000100211b4c zebradzebra_network::peer::connection::Connection$LT$S$C$Tx$GT$::run::
      $u7b$$u7b$closure$u7d$$u7d$::h19990b875630d4ad((null)=ResumeTy @ 0x000070000e6d8960) at connection.rs:273:29
      frame Add a connect command. #16: 0x0000000100353750 zebrad_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h1c95509e633a0dea(self=Pin<&mut core::future::from_generator::GenFuture<generator-0>> @ 0x000070000e6d8ff0, cx=0x000070000e6d91f8) at mod.rs:80:19 frame #17: 0x00000001002741f6 zebrad$LT$tracing_futures..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hf3ee6fef99cc13c9(self=Pin<&mut tracing_futures::Instrumented<core::future::from_generator::GenFuture>> @ 0x000070000e6d9050, cx=0x000070000e6d91f8) at lib.rs:258:9
      frame Ensure GetAddr fingerprinting is not possible. #18: 0x00000001010cb53e zebrad_$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h1f0876d6286c0c96(self=Pin<&mut core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>> @ 0x000070000e6d90b0, cx=0x000070000e6d91f8) at future.rs:119:9 frame #19: 0x000000010110e976 zebrad
      $LT$tracing..instrument..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h51f4e5e7a9f1a74e(self=Pin<&mut tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>>> @ 0x000070000e6d9130, cx=0x000070000e6d91f8) at instrument.rs:151:9
      frame Change message encoding / decoding to be a Tokio codec. #20: 0x000000010114302a zebradtokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h644a0e809bba4c61(ptr=0x0000000107d12f48) at core.rs:173:17 frame #21: 0x00000001010e4535 zebradtokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::hf226087472a41c4c(self=0x0000000107d12f48, f=closure-0 @ 0x000070000e6d9278) at unsafe_cell.rs:14:9
      frame Refactor message serialization as a tokio codec. #22: 0x0000000101142d01 zebradtokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::h9e415263cd1e98c0(self=0x0000000107d12f40, header=0x0000000107d12f10) at core.rs:158:13 frame #23: 0x00000001010fba46 zebradtokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::$u7b$$u7b$closure$u7d$$u7d$::hc237c546787b202c at harness.rs:107:27
      frame Parse inv messages. #24: 0x0000000101152b60 zebradcore::ops::function::FnOnce::call_once::hc422c5d6b5807201((null)=closure-0 @ 0x000070000e6d9398, (null)=<unavailable>) at function.rs:227:5 frame #25: 0x000000010116f51c zebrad
      $LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hbe9869a0021d4266(self=AssertUnwindSafe @ 0x000070000e6d93e0, args=) at panic.rs:322:9
      frame Tracking: parse all message types. #26: 0x00000001010ff0d6 zebradstd::panicking::try::do_call::h8dc826a946c61e83(data="?\x96m\x0e") at panicking.rs:379:40 frame #27: 0x00000001010fec89 zebradstd::panicking::try::h1ec16349b163ce81(f=AssertUnwindSafe @ 0x000070000e6d95a0) at panicking.rs:343:19
      frame Add ecosystem test vectors for message parsing. #28: 0x000000010116f60c zebradstd::panic::catch_unwind::hd82bb5470221ed48(f=AssertUnwindSafe<closure-0> @ 0x000070000e6d9610) at panic.rs:396:14 frame #29: 0x00000001010fb5d1 zebradtokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h86477151a7d93d34(self=Harness<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<Future, alloc::alloc::Global>>>, alloc::sync::Arctokio::runtime::thread_pool::worker::Worker> @ 0x000070000e6d96b0) at harness.rs:89:19
      frame Add an address book to track peer information. #30: 0x0000000101136ff2 zebradtokio::runtime::task::raw::poll::h97b0dd41aebc725b(ptr=NonNull<tokio::runtime::task::core::Header> @ 0x000070000e6d9820) at raw.rs:104:5 frame #31: 0x0000000101618d2e zebradtokio::runtime::task::raw::RawTask::poll::h484811374982c37e(self=RawTask @ 0x000070000e6d9850) at raw.rs:66:18
      frame Transaction naming scheme #32: 0x0000000101657775 zebradtokio::runtime::task::Notified$LT$S$GT$::run::h8bdb0fafd5cb6e78(self=Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> @ 0x000070000e6d9878) at mod.rs:171:9 frame #33: 0x000000010162210c zebradtokio::runtime::thread_pool::worker::Context::run_task::
      $u7b$$u7b$closure$u7d$$u7d$::h139a3b975fade807 at worker.rs:349:13
      frame Implement and use EquihashSolution where appropriate #34: 0x000000010165627e zebradtokio::coop::with_budget::_$u7b$$u7b$closure$u7d$$u7d$::hcc59fdc93e3207b8(cell=0x000000010501f422) at coop.rs:121:9 frame #35: 0x000000010166df7b zebradstd::thread::local::LocalKey$LT$T$GT$::try_with::h2129effad4de7858(self=0x00000001019a35d8, f=closure-0 @ 0x000070000e6d9a90) at local.rs:272:16
      frame Design Sprout note commitment trees. #36: 0x000000010166da4c zebradstd::thread::local::LocalKey$LT$T$GT$::with::hcc57210fbe3db017(self=0x00000001019a35d8, f=<unavailable>) at local.rs:248:9 frame #37: 0x00000001016220b3 zebradtokio::runtime::thread_pool::worker::Context::run_task::hd450d0b0dee086b9 [inlined] tokio::coop::with_budget::h2c4708bfc0b64034(budget=Budget @ 0x000070000e6d9b68, f=closure-0 @ 0x000070000e6d9b70) at coop.rs:114:5
      frame Cleanup and fill out TransactionHash #38: 0x0000000101622097 zebradtokio::runtime::thread_pool::worker::Context::run_task::hd450d0b0dee086b9 [inlined] tokio::coop::budget::had3bb390e47fc55a(f=closure-0 @ 0x000070000e6d9b40) at coop.rs:98 frame #39: 0x0000000101622067 zebradtokio::runtime::thread_pool::worker::Context::run_task::hd450d0b0dee086b9(self=0x000070000e6d9d78, task=Notified<alloc::sync::Arctokio::runtime::thread_pool::worker::Worker> @ 0x000070000e6d9b30, core=0x0000000104f088d0) at worker.rs:348
      frame Use try_bind when building tracing endpoint. #40: 0x0000000101621dd2 zebradtokio::runtime::thread_pool::worker::Context::run::h6285f4142c6e4879(self=0x000070000e6d9d78, core=0x0000000104f088d0) at worker.rs:318:24 frame #41: 0x0000000101621ca5 zebradtokio::runtime::thread_pool::worker::run::$u7b$$u7b$closure$u7d$$u7d$::h1f88d675579854f0 at worker.rs:303:17
      frame Change Services to a PeerServices bitflag. #42: 0x00000001016707d3 zebradtokio::macros::scoped_tls::ScopedKey$LT$T$GT$::set::h678e254b3055849e(self=0x00000001019a3c48, t=0x000070000e6d9d78, f=closure-0 @ 0x000070000e6d9d08) at scoped_tls.rs:61:9 frame #43: 0x0000000101621c68 zebradtokio::runtime::thread_pool::worker::run::hf9a6d2fdc55efa6e(worker=strong=1, weak=0) at worker.rs:300:5
      frame getblocks, headers, getheaders, tx, getdata, notfound #44: 0x0000000101621b71 zebradtokio::runtime::thread_pool::worker::Launch::launch::_$u7b$$u7b$closure$u7d$$u7d$::haba17ebfcc54701d at worker.rs:279:45 frame #45: 0x0000000101624bc8 zebradtokio::runtime::handle::Handle::spawn_blocking::
      $u7b$$u7b$closure$u7d$$u7d$::h52f4b340f4d49a21 at handle.rs:196:17
      frame Add a peer listener. #46: 0x000000010162dd8d zebrad_$LT$tokio..runtime..blocking..task..BlockingTask$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hcc592ac23043352a(self=Pin<&mut tokio::runtime::blocking::task::BlockingTask<closure-0>> @ 0x000070000e6d9e40, _cx=0x000070000e6d9f88) at task.rs:42:21 frame #47: 0x000000010162d7da zebradtokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::$u7b$$u7b$closure$u7d$$u7d$::h642d64634f40a5ea(ptr=0x0000000105f05240) at core.rs:173:17
      frame Implement a PeerSet load-balancer and outgoing connection router. #48: 0x0000000101640a75 zebradtokio::loom::std::unsafe_cell::UnsafeCell$LT$T$GT$::with_mut::he1a7ce483d14cc8b(self=0x0000000105f05240, f=closure-0 @ 0x000070000e6da008) at unsafe_cell.rs:14:9 frame #49: 0x000000010162d67b zebradtokio::runtime::task::core::Core$LT$T$C$S$GT$::poll::he48ae8ae7ebefcee(self=0x0000000105f05240, header=0x0000000105f05210) at core.rs:158:13
      frame Propagate errors from the node's internal Service to the PeerServer. #50: 0x0000000101659296 zebradtokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h5d002f46a0111d31 at harness.rs:107:27 frame #51: 0x00000001016092c0 zebradcore::ops::function::FnOnce::call_once::h79b46edc031a0614((null)=closure-0 @ 0x000070000e6da128, (null)=) at function.rs:227:5
      frame Better version negotiation in the peer connection handshake #52: 0x000000010163054c zebrad_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h527610e2ec650e8d(self=AssertUnwindSafe<closure-0> @ 0x000070000e6da170, _args=<unavailable>) at panic.rs:322:9 frame #53: 0x000000010165a506 zebradstd::panicking::try::do_call::hd1abbb0c3fadf52c(data="@?m\x0e") at panicking.rs:379:40
      frame Add an Abscissa command for DNS seeder information. #54: 0x000000010165a119 zebradstd::panicking::try::h15df02ec242a3124(f=AssertUnwindSafe<closure-0> @ 0x000070000e6da330) at panicking.rs:343:19 frame #55: 0x000000010163143c zebradstd::panic::catch_unwind::h2c3ab54ae552efcd(f=AssertUnwindSafe @ 0x000070000e6da3a0) at panic.rs:396:14
      frame Handle error conversions properly. #56: 0x0000000101658f61 zebradtokio::runtime::task::harness::Harness$LT$T$C$S$GT$::poll::h5c8053b83ae56458(self=Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule> @ 0x000070000e6da440) at harness.rs:89:19 frame #57: 0x0000000101618e72 zebradtokio::runtime::task::raw::poll::h274f5d6871bbe089(ptr=NonNulltokio::runtime::task::core::Header @ 0x000070000e6da5b0) at raw.rs:104:5
      frame Move all git dependencies to published crates. #58: 0x0000000101618d2e zebradtokio::runtime::task::raw::RawTask::poll::h484811374982c37e(self=RawTask @ 0x000070000e6da5e0) at raw.rs:66:18 frame #59: 0x00000001016577a5 zebradtokio::runtime::task::Notified$LT$S$GT$::run::haa7d33e10aaaaf13(self=Notifiedtokio::runtime::blocking::schedule::NoopSchedule @ 0x000070000e6da608) at mod.rs:171:9
      frame Update architecture diagram. #60: 0x000000010161e456 zebradtokio::runtime::blocking::pool::Inner::run::h609078c06565f713(self=0x0000000105f04ca0, worker_thread_id=3) at pool.rs:277:17 frame #61: 0x000000010161e305 zebradtokio::runtime::blocking::pool::Spawner::spawn_thread::
      $u7b$$u7b$closure$u7d$$u7d$::hc9f3b915db1955d6 at pool.rs:257:17
      frame Beginning of peerset implementation. #62: 0x0000000101606404 zebradstd::sys_common::backtrace::__rust_begin_short_backtrace::h21b5fee6238bb59b(f=<unavailable>) at backtrace.rs:125:18 frame #63: 0x000000010163c021 zebradstd::thread::Builder::spawn_unchecked::$u7b$$u7b$closure$u7d$$u7d$::$u7b$$u7b$closure$u7d$$u7d$::h71947a1788d7a4ad at mod.rs:474:17
      frame Remove zebra-reactor. #64: 0x0000000101630591 zebrad_$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h940d5ec5a09ecf00(self=<unavailable>, _args=<unavailable>) at panic.rs:322:9 frame #65: 0x000000010165a475 zebradstd::panicking::try::do_call::h4a4760656e99e6c2(data="\x01") at panicking.rs:379:40
      frame Add peer count data to TimestampData::update trace #66: 0x000000010165a2e3 zebradstd::panicking::try::h40d29b657e3cc91b(f=<unavailable>) at panicking.rs:343:19 frame #67: 0x00000001016314b1 zebradstd::panic::catch_unwind::h74be959446007098(f=) at panic.rs:396:14
      frame Handle inbound connections with PeerConnector. #68: 0x000000010163bf3d zebradstd::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h799872281c27af1d at mod.rs:473:30 frame #69: 0x0000000101609071 zebradcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hb2bdaeb4cdf35cdd((null)=0x0000000105f055a0, (null)=) at function.rs:227:5
      frame Encapsulate all peer-related objects into a network abstraction #70: 0x000000010170ba9d zebradstd::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h1658d06ba2d41cb5 at boxed.rs:1480:9 [opt] frame #71: 0x000000010170ba97 zebradstd::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h747cc083e7349205 at boxed.rs:1480 [opt]
      frame Ascii-fy the command string in message header trace #72: 0x000000010170ba8e zebradstd::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219 at thread.rs:71 [opt] frame #73: 0x00007fff2034a950 libsystem_pthread.dylib_pthread_start + 224
      frame Refactor TimestampCollector into AddressBook #74: 0x00007fff2034647b libsystem_pthread.dylib`thread_start + 15

@teor2345
Copy link
Contributor

teor2345 commented Jan 4, 2021

We're actively working on this code at the moment:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants