Create virtio-net rx packet in one go #2

asias · 2014-10-10T00:10:11Z

From Avi:
'''
Note, this leaves some performance on the table, for two reasons:

packet::append() is less efficient than packet::packet(packet&&, fragment, deleter);
even that is inefficient in that every append allocates a new deleter, and chains it as a linked list into the packet. This both requires one extra allocation per fragment, and walking a linked list when the time comes to kill the packet.

A better approach would be to accumulate the fragments in a vector<>, then create the packet in one go (the closest thing we have is packet(std::vector, deleter) but even better is using two fragment iterators instead of the vector), and creating a deleter that deletes all fragments in one go.
'''

Fixes assert failure during ^C: #0 0x0000003e134348c7 in raise () from /lib64/libc.so.6 #1 0x0000003e1343652a in abort () from /lib64/libc.so.6 #2 0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_ at /usr/include/boost/intrusive/list.hpp:1263 #5 0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71 #6 reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287 #7 0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974 #8 ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86 #9 ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96

Records allocation sites and sizes so that it's later possible to see what kind of objects are taking up space on the heap. Adds 8 bytes of memory overhead per live object when compiled-in. Not compiled-in by default. To do so, configure with SEASTAR_HEAPPROF: ./configure.py --cflags=-DSEASTAR_HEAPPROF To enable recording allocation sites, start seastar application with the --heapprof flag: build/release/memcached --heapprof It is possible to enable recording in run-time using, for instance, a RESTful API. scylla-gdb.py support will follow, with text-mode tree presentation and dumping to flamegraphs. Flamegraph example: https://cloud.githubusercontent.com/assets/283695/18888026/f69fc4e6-84f6-11e6-9b7b-305667d30f52.png Text example: (gdb) scylla heapprof -r All (275954028, #12783) |-- void* memory::cpu_pages::allocate_large_and_trim<memory::cpu_pages::allocate_large_aligned(unsigned int, unsigned int)::{lambda(unsigned int, unsigned int)#1}>(unsigned int, memory::cpu_pages::allocate_large_aligned(unsigned int, unsigned int)::{lambda(unsigned int, unsigned int)#1}) + 169 (268959744, #5) | memory::allocate_large_aligned(unsigned long, unsigned long) + 87 | |-- logalloc::segment_zone::segment_zone() + 291 (268435456, #1) | | logalloc::segment_pool::allocate_segment() + 413 | | logalloc::segment_pool::segment_pool() + 296 | | __tls_init.part.787 + 72 | | logalloc::region_group::release_requests() + 1333 | | logalloc::region_group::add(logalloc::region_group*) + 514 | | database::database(db::config const&) + 4246 | | (...) | | | \-- memory::allocate_aligned(unsigned long, unsigned long) + 13 (524288, #4) | |-- memalign + 9 (262144, #2) | | db::commitlog::segment_manager::acquire_buffer(unsigned long) + 90 | | db::commitlog::segment::new_buffer(unsigned long) + 113 | | db::commitlog::segment::allocate(utils::UUID const&, shared_ptr<db::commitlog::entry_writer>) + 2347 | | |-- db::commitlog::add_entry(utils::UUID const&, commitlog_entry_writer const&) + 1620 (131072, #1) | | | database::do_apply(lw_shared_ptr<schema const>, frozen_mutation const&) + 182 | | | database::apply(lw_shared_ptr<schema const>, frozen_mutation const&) + 235 | | | service::storage_proxy::mutate_locally(lw_shared_ptr<schema const> const&, frozen_mutation const&) + 434 Message-Id: <[email protected]>

When _snd.window == 0, it is not allowed to send a tcp segment with more than 1 byte of data for zero window probe. However, it is legal to send an empty segment, even if _snd.window == 0. When a seastar tcp client connects to a tcp server: With: void do_syn_sent() { _state = SYN_SENT; _snd.syn_tx_time = clock_type::now(); + print("In do_syn_sent: _snd.window = %ld\n", _snd.window); // Send <SYN> to remote output(); } I saw: In do_syn_sent: _snd.window = 0 So the callchain is: do_syn_sent() (with a _snd.window == 0)-> output() -> get_packet() -> output_one -> get the assert client: net/tcp.hh:1691: void seastar::net::tcp<InetTraits>::tcb::output_one(bool) [with InetTraits = seastar::net::ipv4_traits]: Assertion `(_snd.window > 0) || ((_snd.window == 0) && (len == 1))' failed. $ addr2line -Cfpi -e ./client 0x000000000041fa8a 0x000000000041fb5c 0x000000000041fc13 /lib64/libpthread.so.0+0x000000000001241f /lib64/libc.so.6+0x00000000000369fa /lib64/libc.so.6+0x00000000000387ff /lib64/libc.so.6+0x000000000002f0d9 /lib64/libc.so.6+0x000000000002f151 0x00000000005d0cc7 0x00000000005d41f9 0x00000000005d46e8 0x00000000005cc2f3 0x00000000005a7ad0 0x00000000005a7bfb 0x000000000052c3b2 0x000000000052d752 0x000000000041a814 0x000000000046b47e 0x00000000004fb9f2 0x00000000004fc642 0x0000000000418e1d /lib64/libc.so.6+0x0000000000020889 0x0000000000418fc9 seastar::backtrace_buffer::append_backtrace() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:378 (inlined by) print_with_backtrace at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:399 seastar::print_with_backtrace(char const*) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:406 sigabrt_action at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3914 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3896 (inlined by) _FUN at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3892 ?? ??:0 ?? ??:0 ?? ??:0 ?? ??:0 ?? ??:0 seastar::net::tcp<seastar::net::ipv4_traits>::tcb::output_one(bool) at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:1691 (discriminator 2) seastar::net::tcp<seastar::net::ipv4_traits>::tcb::get_packet() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:2046 (inlined by) seastar::net::tcp<seastar::net::ipv4_traits>::tcp(seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>&)::{lambda()#2}::operator()() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:770 std::_Function_handler<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> (), seastar::net::tcp<seastar::net::ipv4_traits>::tcp(seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>&)::{lambda()#2}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/7/bits/std_function.h:302 std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706 (inlined by) seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>::register_packet_provider(std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>)::{lambda()#1}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.hh:424 (inlined by) std::_Function_handler<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> (), seastar::net::ipv4_l4<(seastar::net::ip_protocol_num)6>::register_packet_provider(std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/7/bits/std_function.h:302 std::function<std::experimental::fundamentals_v1::optional<seastar::net::ipv4_traits::l4packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706 (inlined by) seastar::net::ipv4::get_packet() at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.cc:302 operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/ip.cc:54 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302 std::function<std::experimental::fundamentals_v1::optional<seastar::net::l3_protocol::l3packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:246 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302 std::function<std::experimental::fundamentals_v1::optional<seastar::net::packet> ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706 (inlined by) seastar::net::qp::poll_tx() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:68 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/net/net.cc:89 (inlined by) poll at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.hh:1147 seastar::reactor::poll_once() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3254 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3150 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:302 std::function<bool ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706 (inlined by) seastar::reactor::run() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3175 seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:185 seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:113 main at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/client.cc:167 Message-Id: <5b6fddc098a1c936dd7c6833e33059d0264bb3b7.1526611968.git.asias@scylladb.com>

The receive window size should be adjusted according to the max receiver buffer. Fix the following abort. Without reducing the receive window, the sender has no back pressure, if the application does not read the data, the _rcv.data grows unbounded, when the application read again, all the packet in _rcv.data will be merged into a single packet with tons of fragments and cause the assert. In seastar::net::tcp<seastar::net::ipv4_traits>::tcb::read(): for (auto&& q : _rcv.data) { p.append(std::move(q)); } server: /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:144: static void* seastar::net::packet::impl::operator new(size_t, size_t): Assertion `nr_frags == uint16_t(nr_frags)' failed. Aborting on shard 0. Backtrace: 0x000000000041fafa 0x000000000041fbcc 0x000000000041fc83 /lib64/libpthread.so.0+0x000000000001241f /lib64/libc.so.6+0x00000000000369fa /lib64/libc.so.6+0x00000000000387ff /lib64/libc.so.6+0x000000000002f0d9 /lib64/libc.so.6+0x000000000002f151 0x00000000004b89dd 0x00000000005ab01e 0x00000000005cbc6a 0x00000000005dbfc1 0x00000000005db9b8 0x0000000000570986 0x0000000000570be5 0x000000000069859a 0x00000000006a16e5 0x000000000041d136 0x000000000041d422 0x000000000046b527 0x00000000004fba62 0x0000000000418e77 /lib64/libc.so.6+0x0000000000020889 0x0000000000419039 $ addr2line -Cfpi -e ./server 0x000000000041fafa 0x000000000041fbcc 0x000000000041fc83 /lib64/libpthread.so.0+0x000000000001241f /lib64/libc.so.6+0x00000000000369fa /lib64/libc.so.6+0x00000000000387ff /lib64/libc.so.6+0x000000000002f0d9 /lib64/libc.so.6+0x000000000002f151 0x00000000004b89dd 0x00000000005ab01e 0x00000000005cbc6a 0x00000000005dbfc1 0x00000000005db9b8 0x0000000000570986 0x0000000000570be5 0x000000000069859a 0x00000000006a16e5 0x000000000041d136 0x000000000041d422 0x000000000046b527 0x00000000004fba62 0x0000000000418e77 /lib64/libc.so.6+0x0000000000020889 0x0000000000419039 seastar::backtrace_buffer::append_backtrace() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:378 (inlined by) print_with_backtrace at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:399 seastar::print_with_backtrace(char const*) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:406 sigabrt_action at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3914 (inlined by) operator() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3896 (inlined by) _FUN at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3892 ?? ??:0 ?? ??:0 ?? ??:0 ?? ??:0 ?? ??:0 seastar::net::packet::impl::operator new(unsigned long, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:144 (inlined by) seastar::net::packet::impl::allocate(unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:117 (inlined by) seastar::net::packet::impl::copy(seastar::net::packet::impl*, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:121 (inlined by) seastar::net::packet::impl::allocate_if_needed(std::unique_ptr<seastar::net::packet::impl, std::default_delete<seastar::net::packet::impl> >, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:141 seastar::net::packet::append(seastar::net::packet&&) at /home/asias/src/cloudius-systems/scylla/seastar/net/packet.hh:490 seastar::net::tcp<seastar::net::ipv4_traits>::tcb::read() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:1760 (inlined by) seastar::net::tcp<seastar::net::ipv4_traits>::connection::read() at /home/asias/src/cloudius-systems/scylla/seastar/net/tcp.hh:682 (inlined by) seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/net/native-stack-impl.hh:145 (inlined by) seastar::apply_helper<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}, std::tuple<>&&, std::integer_sequence<unsigned long> >::apply({lambda()#2}&&, std::tuple<>) at /home/asias/src/cloudius-systems/scylla/seastar/core/apply.hh:36 (inlined by) auto seastar::apply<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}>(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&, std::tuple<>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/apply.hh:44 (inlined by) seastar::future<seastar::temporary_buffer<char> > seastar::futurize<seastar::future<seastar::temporary_buffer<char> > >::apply<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}>(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&, std::tuple<>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1377 seastar::future<seastar::temporary_buffer<char> > seastar::future<>::then<seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}, seastar::future<seastar::temporary_buffer<char> > >(seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get()::{lambda()#2}&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:943 seastar::net::native_connected_socket_impl<seastar::net::tcp<seastar::net::ipv4_traits> >::native_data_source_impl::get() at /home/asias/src/cloudius-systems/scylla/seastar/net/native-stack-impl.hh:149 seastar::data_source::get() at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream.hh:67 (inlined by) seastar::input_stream<char>::read_exactly_part(unsigned long, seastar::temporary_buffer<char>, unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream-impl.hh:164 seastar::input_stream<char>::read_exactly(unsigned long) at /home/asias/src/cloudius-systems/scylla/seastar/core/iostream-impl.hh:191 tcp_echo_server::connection::process() at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:86 (inlined by) tcp_echo_server::start()::{lambda()#1}::operator()() const::{lambda(seastar::connected_socket, {lambda()#1}::socket_address)#1}::operator()(seastar, seastar::connected_socket)::{lambda()#2}::operator()() const at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:113 seastar::internal::do_until_state<tcp_echo_server::start()::{lambda()#1}::operator()() const::{lambda(seastar::connected_socket, seastar::socket_address)#1}::operator()({lambda()#1}, seastar::connected_socket)::{lambda()#1}, tcp_echo_server::start()::{lambda()#1}::operator()() const::seastar::socket_address::operator()({lambda()#1}, seastar::connected_socket)::{lambda()#2}>::run_and_dispose() at /home/asias/src/cloudius-systems/scylla/seastar/core/future-util.hh:463 seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:2597 seastar::reactor::run_some_tasks() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3009 seastar::reactor::run_some_tasks() at /usr/include/c++/7/chrono:377 (inlined by) seastar::reactor::run() at /home/asias/src/cloudius-systems/scylla/seastar/core/reactor.cc:3156 seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/asias/src/cloudius-systems/scylla/seastar/core/app-template.cc:185 main at /home/asias/src/cloudius-systems/scylla/seastar/seastar_example_from_shanshanpt/server.cc:179 (discriminator 5) Message-Id: <06a6c0ccb82fd4fdde6f56a112a8eb6f659ee8ab.1526611968.git.asias@scylladb.com>

Hb web socket

This reverts commit 33406cf. It introduces memory leaks: Direct leak of 24 byte(s) in 1 object(s) allocated from: #0 0x7fb773b389d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) #1 0x108f0d4 in seastar::reactor::poller::~poller() ../src/core/reactor.cc:2879 #2 0x11c1e59 in std::experimental::fundamentals_v1::_Optional_base<seastar::reactor::poller, true>::~_Optional_base() /usr/include/c++/9/experimental/optional:288 #3 0x118f2d7 in std::experimental::fundamentals_v1::optional<seastar::reactor::poller>::~optional() /usr/include/c++/9/experimental/optional:491 #4 0x108c5a5 in seastar::reactor::run() ../src/core/reactor.cc:2587 #5 0xf1a822 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) ../src/core/app-template.cc:199 #6 0xf1885d in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) ../src/core/app-template.cc:115 #7 0xeb2735 in operator() ../src/testing/test_runner.cc:72 #8 0xebb342 in _M_invoke /usr/include/c++/9/bits/std_function.h:300 #9 0xf3d8b0 in std::function<void ()>::operator()() const /usr/include/c++/9/bits/std_function.h:690 #10 0x1034c72 in seastar::posix_thread::start_routine(void*) ../src/core/posix.cc:52 #11 0x7fb7738804e1 in start_thread /usr/src/debug/glibc-2.30-13-g919af705ee/nptl/pthread_create.c:479 Reported-by: Rafael Avila de Espindola <[email protected]>

…o_with Fixes failures in debug mode: ``` $ build/debug/tests/unit/closeable_test -l all -t deferred_close_test WARNING: debug mode. Not for benchmarking or production random-seed=3064133628 Running 1 test case... Entering test module "../../tests/unit/closeable_test.cc" ../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test" ../../src/testing/seastar_test.cc(43): info: check true has passed ==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! terminate called after throwing an instance of 'seastar::broken_promise' what(): broken promise ==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592) False positive error reports may follow For details see google/sanitizers#189 ================================================================= ==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0) #0 0x7fbf45f39d0b (/lib64/libasan.so.6+0xb3d0b) #1 0x7fbf45f57d4e (/lib64/libasan.so.6+0xd1d4e) #2 0x7fbf45f3e724 (/lib64/libasan.so.6+0xb8724) #3 0x7fbf45eb3e5b (/lib64/libasan.so.6+0x2de5b) #4 0x7fbf45eb51e8 (/lib64/libasan.so.6+0x2f1e8) #5 0x7fbf45eb7694 (/lib64/libasan.so.6+0x31694) #6 0x7fbf45f39398 (/lib64/libasan.so.6+0xb3398) #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b) #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116 #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121 #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810) #11 0x7fbf44bcd897 (/lib64/libgcc_s.so.1+0xf897) #12 0x7fbf44bcea5f (/lib64/libgcc_s.so.1+0x10a5f) #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8) #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148 #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb) #16 0x7fbf45e10a21 (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21) #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0) #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094) #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921) #20 0x7fbf45e5eae1 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1) #21 0x7fbf45e5ed31 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31) #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547) #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618) #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77 #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65 #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1) #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd) ``` Signed-off-by: Benny Halevy <[email protected]> Message-Id: <[email protected]>

This is the continuation of the patch scylladb#2 -- the 2-step initialization of io-groups is patched to use one container object instead of two. Signed-off-by: Pavel Emelyanov <[email protected]>

When we enable the sanitizer, we get following error while running iotune: ==86505==ERROR: LeakSanitizer: detected memory leaks Direct leak of 4096 byte(s) in 1 object(s) allocated from: #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7) scylladb#1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16 scylladb#2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21 scylladb#3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28 scylladb#4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13 scylladb#5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20 scylladb#6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9 scylladb#7 0x7f30ff8547bf (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c) SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s). This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error. And this is needed to test Scylla machine image with debug mode binary, since it tries to run iotune with the sanitizer and fails.

When we enable the sanitizer, we get following error while running iotune: ==86505==ERROR: LeakSanitizer: detected memory leaks Direct leak of 4096 byte(s) in 1 object(s) allocated from: #0 0x5701b8 in aligned_alloc (/home/syuu/seastar.2/build/sanitize/apps/iotune/iotune+0x5701b8) (BuildId: 411f9852d64ed8982d5b33d02489b5932d92b8b7) #1 0x6d0813 in seastar::filesystem_has_good_aio_support(seastar::basic_sstring<char, unsigned int, 15u, true>, bool) /home/syuu/seastar.2/src/core/fsqual.cc:74:16 #2 0x5bcd0d in main::$_0::operator()() const::'lambda'()::operator()() const /home/syuu/seastar.2/apps/iotune/iotune.cc:742:21 #3 0x5bb1f1 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/syuu/seastar.2/include/seastar/core/future.hh:2118:28 #4 0x5bb039 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/syuu/seastar.2/include/seastar/core/thread.hh:258:13 #5 0x5bb039 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/syuu/seastar.2/include/seastar/util/noncopyable_function.hh:124:20 #6 0x8e0a77 in seastar::thread_context::main() /home/syuu/seastar.2/src/core/thread.cc:299:9 #7 0x7f30ff8547bf (/lib64/libc.so.6+0x547bf) (BuildId: 85c438f4ff93e21675ff174371c9c583dca00b2c) SUMMARY: AddressSanitizer: 4096 byte(s) leaked in 1 allocation(s). This is because we don't free buffer which allocated at filesystem_has_good_aio_support(), we should free it to avoid such error. And this is needed to test Scylla machine image with debug mode binary, since it tries to run iotune with the sanitizer and fails. Closes #1284

================================================================= ==93534==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f676abad010 in thread T0 #0 0x7f677fedfdc2 in __interceptor_free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 scylladb#1 0x7f6776e81776 in operator() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1242 scylladb#2 0x7f6776e884a4 in ~deferred_action /home/sitano/Projects/seastar/include/seastar/util/defer.hh:67 scylladb#3 0x7f6776e82d31 in try_create_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1253 scylladb#4 0x7f6776e83e39 in detect_io_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1288 scylladb#5 0x7f6776e85113 in seastar::reactor_backend_selector::available() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:2039 scylladb#6 0x7f67773be492 in seastar::reactor_options::reactor_options(seastar::program_options::option_group*) /home/sitano/Projects/seastar/src/core/reactor.cc:3782 scylladb#7 0x7f6776f04c32 in seastar::app_template::seastar_options::seastar_options() /home/sitano/Projects/seastar/src/core/app-template.cc:70 scylladb#8 0x7f6776f04040 in seastar_options_from_config /home/sitano/Projects/seastar/src/core/app-template.cc:58 scylladb#9 0x7f6776f0704f in seastar::app_template::app_template(seastar::app_template::config) /home/sitano/Projects/seastar/src/core/app-template.cc:108 scylladb#10 0x55a5ff597a6e in main /home/sitano/Projects/seastar/demos/hello-world.cc:30 scylladb#11 0x7f676f03984f (/usr/lib/libc.so.6+0x2384f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#12 0x7f676f039909 in __libc_start_main (/usr/lib/libc.so.6+0x23909) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#13 0x55a5ff597614 in _start (/home/sitano/Projects/seastar/build/debug/demos/hello-world_demo+0xf0614) (BuildId: cbf8a43d1565f3bf0a48cdfb43a11ce16d7a9bc1) Address 0x7f676abad010 is a wild pointer inside of access range of size 0x000000000001. SUMMARY: AddressSanitizer: bad-free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 in __interceptor_free ==93534==ABORTING

in main(), we creates an instance of `http_server_control` using new, but we never destroy it. this is identified by ASan ``` ==2190125==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3 #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27 #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl ude/c++/14/bits/invoke.h:61:14 #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_ 64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14 #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, . ..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14 #5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../ ../include/c++/14/tuple:2936:14 #6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea star/core/future.hh:2005:28 #7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0: :operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13 #8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable _function.hh:129:20 #9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16 #10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9 #11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43 #12 0x7f5d72f8a18f (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06) ``` so, in this change, let's hold it using a smart pointer, so we can destroy it when it leaves the lexical scope. Signed-off-by: Kefu Chai <[email protected]> Closes #2224

asias closed this as completed Oct 28, 2014

asias mentioned this issue Oct 29, 2014

apachebench hangs on httpd #3

Closed

slivne mentioned this issue Jan 19, 2015

support running on hyperthreads #15

Closed

vladzcloudius mentioned this issue Jan 28, 2015

DHCP timeout on AWS guest #20

Open

gleb-cloudius mentioned this issue Oct 18, 2015

memcached crashes on setting first key #64

Closed

shivshankardayal mentioned this issue Nov 30, 2015

seastar segfaults on Ubuntu 15.10 #81

Closed

aaalgo mentioned this issue Jan 2, 2016

httpd with DPDK segmentation fault #99

Open

npmadhu mentioned this issue Jan 23, 2016

allocator_test fails - alloc-dealloc-mismatch #111

Open

kthommandra mentioned this issue Aug 24, 2016

Core dump in scylla-server with DHCP in DPDK mode #184

Closed

tedqian mentioned this issue Sep 28, 2016

seastar segfaults in tcp_test on Ubuntu 16.04 LTS #198

Open

amnonh mentioned this issue Nov 21, 2016

Reactor unregister its metrics after the metrics shutdown #218

Closed

asimarora mentioned this issue Nov 25, 2017

Core Dump while running sample program #345

Closed

nlfox mentioned this issue Jan 13, 2018

Memcached bad performance (both using DPDK and posix network stack) #383

Open

frank8989 mentioned this issue Feb 26, 2018

Task block detector deadlocks with exception throwing #298

Open

tedqian mentioned this issue Mar 12, 2018

httpd print “std::system_error (error system:103, Software caused connection abort),” when seawreck run with multiple cores and connections #418

Closed

joeylichang mentioned this issue Mar 30, 2018

what(): network stack native not registeredterminate called recursively #423

Closed

closertotheheart mentioned this issue Jan 30, 2019

1/47 Test #1: Seastar.dist.consumer .........................***Failed 86.95 sec #596

Closed

ashaffer pushed a commit to ashaffer/seastar that referenced this issue Jul 17, 2019

Merge pull request scylladb#2 from HippoBaro/hb_web_socket

f180a36

Hb web socket

loginhel mentioned this issue May 16, 2020

how to run tcp_demo #731

Open

hiseyxhp mentioned this issue Jun 23, 2020

when rpc client closed by server, the cpu 100% #762

Open

Howe2015 mentioned this issue Aug 31, 2020

http server dump #781

Open

vladzcloudius mentioned this issue Mar 23, 2021

perftune.py: make RPS CPU set to be a full CPU set #860

Open

zhanglistar mentioned this issue Jan 21, 2022

echo_demo core dump #1003

Closed

vladzcloudius added a commit to vladzcloudius/seastar that referenced this issue Aug 19, 2022

perftune.py: work in progress scylladb#2

aa34f2f

jordan-at-cy mentioned this issue Mar 4, 2023

Segmentation fault when running demos/tpc_demo.cc #1526

Open

nlescoua mentioned this issue Mar 28, 2023

Abort called when writing a temporary_buffer to a file_output_stream #1581

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create virtio-net rx packet in one go #2

Create virtio-net rx packet in one go #2

asias commented Oct 10, 2014

Create virtio-net rx packet in one go #2

Create virtio-net rx packet in one go #2

Comments

asias commented Oct 10, 2014