Incorrect default processor count logic #8

nyh · 2014-11-11T12:00:35Z

When Seastar is run on a machine with hyperthreads with default parameters, it fails with:

terminate called after throwing an instance of 'std::runtime_error'
  what():  insufficient processing units

The problem is that if you have, for example, 12 cores and 24 hyperthreads, configuration.cpus is set, by default, to 24, yet resource::allocate() (resource.cc) checks, and fails with the above exception, if that is higher than 12.

The text was updated successfully, but these errors were encountered:

nyh · 2014-11-16T15:35:29Z

The default memory setup has the same bug. In the hyperthreaded setup described above, if I give the application "-c 1", the above failure is circumvented, but then we get to this failure:

core/resource.cc:71: std::vector<resource::cpu> resource::allocate(resource::configuration): Assertion `!remain' failed.

This second message is avoided if memory is also specified non-default, e.g., "-c 1 -m 4G".

Fixes assert failure during ^C: #0 0x0000003e134348c7 in raise () from /lib64/libc.so.6 #1 0x0000003e1343652a in abort () from /lib64/libc.so.6 #2 0x0000003e1342d46d in __assert_fail_base () from /lib64/libc.so.6 #3 0x0000003e1342d522 in __assert_fail () from /lib64/libc.so.6 #4 0x0000000000409a7c in boost::intrusive::list_impl<boost::intrusive::mhtraits<timer, boost::intrusive::list_ at /usr/include/boost/intrusive/list.hpp:1263 #5 0x00000000004881cc in iterator_to (this=<optimized out>, value=...) at core/timer-set.hh:71 #6 reactor::del_timer (this=<optimized out>, tmr=tmr@entry=0x60000005cda8) at core/reactor.cc:287 #7 0x00000000004682a5 in ~timer (this=0x60000005cda8, __in_chrg=<optimized out>) at ./core/reactor.hh:974 #8 ~resolution (this=0x60000005cd90, __in_chrg=<optimized out>) at net/arp.hh:86 #9 ~pair (this=0x60000005cd88, __in_chrg=<optimized out>) at /usr/include/c++/4.9.2/bits/stl_pair.h:96

penberg · 2015-01-19T17:59:49Z

AFAICT, this is fixed by 93ae11c

This reverts commit 33406cf. It introduces memory leaks: Direct leak of 24 byte(s) in 1 object(s) allocated from: #0 0x7fb773b389d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) #1 0x108f0d4 in seastar::reactor::poller::~poller() ../src/core/reactor.cc:2879 #2 0x11c1e59 in std::experimental::fundamentals_v1::_Optional_base<seastar::reactor::poller, true>::~_Optional_base() /usr/include/c++/9/experimental/optional:288 #3 0x118f2d7 in std::experimental::fundamentals_v1::optional<seastar::reactor::poller>::~optional() /usr/include/c++/9/experimental/optional:491 #4 0x108c5a5 in seastar::reactor::run() ../src/core/reactor.cc:2587 #5 0xf1a822 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) ../src/core/app-template.cc:199 #6 0xf1885d in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) ../src/core/app-template.cc:115 #7 0xeb2735 in operator() ../src/testing/test_runner.cc:72 #8 0xebb342 in _M_invoke /usr/include/c++/9/bits/std_function.h:300 #9 0xf3d8b0 in std::function<void ()>::operator()() const /usr/include/c++/9/bits/std_function.h:690 #10 0x1034c72 in seastar::posix_thread::start_routine(void*) ../src/core/posix.cc:52 #11 0x7fb7738804e1 in start_thread /usr/src/debug/glibc-2.30-13-g919af705ee/nptl/pthread_create.c:479 Reported-by: Rafael Avila de Espindola <[email protected]>

…o_with Fixes failures in debug mode: ``` $ build/debug/tests/unit/closeable_test -l all -t deferred_close_test WARNING: debug mode. Not for benchmarking or production random-seed=3064133628 Running 1 test case... Entering test module "../../tests/unit/closeable_test.cc" ../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test" ../../src/testing/seastar_test.cc(43): info: check true has passed ==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! terminate called after throwing an instance of 'seastar::broken_promise' what(): broken promise ==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592) False positive error reports may follow For details see google/sanitizers#189 ================================================================= ==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0) #0 0x7fbf45f39d0b (/lib64/libasan.so.6+0xb3d0b) #1 0x7fbf45f57d4e (/lib64/libasan.so.6+0xd1d4e) #2 0x7fbf45f3e724 (/lib64/libasan.so.6+0xb8724) #3 0x7fbf45eb3e5b (/lib64/libasan.so.6+0x2de5b) #4 0x7fbf45eb51e8 (/lib64/libasan.so.6+0x2f1e8) #5 0x7fbf45eb7694 (/lib64/libasan.so.6+0x31694) #6 0x7fbf45f39398 (/lib64/libasan.so.6+0xb3398) #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b) #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116 #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121 #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810) #11 0x7fbf44bcd897 (/lib64/libgcc_s.so.1+0xf897) #12 0x7fbf44bcea5f (/lib64/libgcc_s.so.1+0x10a5f) #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8) #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148 #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb) #16 0x7fbf45e10a21 (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21) #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0) #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094) #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921) #20 0x7fbf45e5eae1 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1) #21 0x7fbf45e5ed31 (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31) #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547) #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618) #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77 #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65 #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1) #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd) ``` Signed-off-by: Benny Halevy <[email protected]> Message-Id: <[email protected]>

httpd: add status_type 409

================================================================= ==93534==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f676abad010 in thread T0 #0 0x7f677fedfdc2 in __interceptor_free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 scylladb#1 0x7f6776e81776 in operator() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1242 scylladb#2 0x7f6776e884a4 in ~deferred_action /home/sitano/Projects/seastar/include/seastar/util/defer.hh:67 scylladb#3 0x7f6776e82d31 in try_create_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1253 scylladb#4 0x7f6776e83e39 in detect_io_uring /home/sitano/Projects/seastar/src/core/reactor_backend.cc:1288 scylladb#5 0x7f6776e85113 in seastar::reactor_backend_selector::available() /home/sitano/Projects/seastar/src/core/reactor_backend.cc:2039 scylladb#6 0x7f67773be492 in seastar::reactor_options::reactor_options(seastar::program_options::option_group*) /home/sitano/Projects/seastar/src/core/reactor.cc:3782 scylladb#7 0x7f6776f04c32 in seastar::app_template::seastar_options::seastar_options() /home/sitano/Projects/seastar/src/core/app-template.cc:70 scylladb#8 0x7f6776f04040 in seastar_options_from_config /home/sitano/Projects/seastar/src/core/app-template.cc:58 scylladb#9 0x7f6776f0704f in seastar::app_template::app_template(seastar::app_template::config) /home/sitano/Projects/seastar/src/core/app-template.cc:108 scylladb#10 0x55a5ff597a6e in main /home/sitano/Projects/seastar/demos/hello-world.cc:30 scylladb#11 0x7f676f03984f (/usr/lib/libc.so.6+0x2384f) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#12 0x7f676f039909 in __libc_start_main (/usr/lib/libc.so.6+0x23909) (BuildId: 2f005a79cd1a8e385972f5a102f16adba414d75e) scylladb#13 0x55a5ff597614 in _start (/home/sitano/Projects/seastar/build/debug/demos/hello-world_demo+0xf0614) (BuildId: cbf8a43d1565f3bf0a48cdfb43a11ce16d7a9bc1) Address 0x7f676abad010 is a wild pointer inside of access range of size 0x000000000001. SUMMARY: AddressSanitizer: bad-free /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52 in __interceptor_free ==93534==ABORTING

in main(), we creates an instance of `http_server_control` using new, but we never destroy it. this is identified by ASan ``` ==2190125==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3 #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27 #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl ude/c++/14/bits/invoke.h:61:14 #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_ 64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14 #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, . ..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14 scylladb#5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../ ../include/c++/14/tuple:2936:14 scylladb#6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea star/core/future.hh:2005:28 scylladb#7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0: :operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13 scylladb#8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable _function.hh:129:20 scylladb#9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16 scylladb#10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9 scylladb#11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43 scylladb#12 0x7f5d72f8a18f (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06) ``` so, in this change, let's hold it using a smart pointer, so we can destroy it when it leaves the lexical scope. Signed-off-by: Kefu Chai <[email protected]>

in main(), we creates an instance of `http_server_control` using new, but we never destroy it. this is identified by ASan ``` ==2190125==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x55e21cf487bd in operator new(unsigned long) /home/kefu/dev/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:86:3 #1 0x55e21cf6cf31 in main::$_0::operator()() const::'lambda'()::operator()() const /home/kefu/dev/seastar/apps/httpd/main.cc:121:27 #2 0x55e21cf6b4cc in int std::__invoke_impl<int, main::$_0::operator()() const::'lambda'()>(std::__invoke_other, main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../incl ude/c++/14/bits/invoke.h:61:14 #3 0x55e21cf6b46c in std::__invoke_result<main::$_0::operator()() const::'lambda'()>::type std::__invoke<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&) /usr/lib/gcc/x86_ 64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96:14 #4 0x55e21cf6b410 in decltype(auto) std::__apply_impl<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&, std::integer_sequence<unsigned long, . ..>) /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921:14 #5 0x55e21cf6b3b2 in decltype(auto) std::apply<main::$_0::operator()() const::'lambda'(), std::tuple<>>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /usr/lib/gcc/x86_64-redhat-linux/14/../../../ ../include/c++/14/tuple:2936:14 #6 0x55e21cf6b283 in seastar::future<int> seastar::futurize<int>::apply<main::$_0::operator()() const::'lambda'()>(main::$_0::operator()() const::'lambda'()&&, std::tuple<>&&) /home/kefu/dev/seastar/include/sea star/core/future.hh:2005:28 #7 0x55e21cf6b043 in seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0: :operator()() const::'lambda'()&&)::'lambda'()::operator()() const /home/kefu/dev/seastar/include/seastar/core/thread.hh:260:13 #8 0x55e21cf6ae74 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<main::$_0::operator()() const::'lambda'()>::type>::type seastar::async<main::$_0::operator()() const::'lambda'()>(seastar::thread_attributes, main::$_0::operator()() const::'lambda'()&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/kefu/dev/seastar/include/seastar/util/noncopyable _function.hh:129:20 #9 0x7f5d757a0fb3 in seastar::noncopyable_function<void ()>::operator()() const /home/kefu/dev/seastar/include/seastar/util/noncopyable_function.hh:215:16 #10 0x7f5d75ef5611 in seastar::thread_context::main() /home/kefu/dev/seastar/src/core/thread.cc:311:9 #11 0x7f5d75ef50eb in seastar::thread_context::s_main(int, int) /home/kefu/dev/seastar/src/core/thread.cc:287:43 #12 0x7f5d72f8a18f (/lib64/libc.so.6+0x5a18f) (BuildId: b098f1c75a76548bb230d8f551eae07a2aeccf06) ``` so, in this change, let's hold it using a smart pointer, so we can destroy it when it leaves the lexical scope. Signed-off-by: Kefu Chai <[email protected]> Closes #2224

nyh mentioned this issue Jan 19, 2015

support running on hyperthreads #15

Closed

avikivity closed this as completed Jan 27, 2015

vladzcloudius mentioned this issue Jan 28, 2015

DHCP timeout on AWS guest #20

Open

gleb-cloudius mentioned this issue Oct 18, 2015

memcached crashes on setting first key #64

Closed

shivshankardayal mentioned this issue Nov 30, 2015

seastar segfaults on Ubuntu 15.10 #81

Closed

aaalgo mentioned this issue Jan 2, 2016

httpd with DPDK segmentation fault #99

Open

kthommandra mentioned this issue Aug 24, 2016

Core dump in scylla-server with DHCP in DPDK mode #184

Closed

amnonh mentioned this issue Nov 21, 2016

Reactor unregister its metrics after the metrics shutdown #218

Closed

asimarora mentioned this issue Nov 25, 2017

Core Dump while running sample program #345

Closed

nlfox mentioned this issue Jan 13, 2018

Memcached bad performance (both using DPDK and posix network stack) #383

Open

frank8989 mentioned this issue Feb 26, 2018

Task block detector deadlocks with exception throwing #298

Open

joeylichang mentioned this issue Mar 30, 2018

what(): network stack native not registeredterminate called recursively #423

Closed

closertotheheart mentioned this issue Jan 30, 2019

1/47 Test #1: Seastar.dist.consumer .........................***Failed 86.95 sec #596

Closed

loginhel mentioned this issue May 16, 2020

how to run tcp_demo #731

Open

hiseyxhp mentioned this issue Jun 23, 2020

when rpc client closed by server, the cpu 100% #762

Open

Howe2015 mentioned this issue Aug 31, 2020

http server dump #781

Open

jcsp pushed a commit to jcsp/seastar that referenced this issue Aug 5, 2021

Merge pull request scylladb#8 from dimitriscruz/conflict-code

8cdc3c1

httpd: add status_type 409

zhanglistar mentioned this issue Jan 21, 2022

echo_demo core dump #1003

Closed

nlescoua mentioned this issue Mar 28, 2023

Abort called when writing a temporary_buffer to a file_output_stream #1581

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect default processor count logic #8

Incorrect default processor count logic #8

nyh commented Nov 11, 2014

nyh commented Nov 16, 2014

penberg commented Jan 19, 2015

Incorrect default processor count logic #8

Incorrect default processor count logic #8

Comments

nyh commented Nov 11, 2014

nyh commented Nov 16, 2014

penberg commented Jan 19, 2015