Skip to content
This repository has been archived by the owner on Jun 23, 2022. It is now read-only.

fix(asan): fix heap-use-after-free in perf_counters #773

Merged
merged 8 commits into from
Mar 5, 2021

Conversation

levy5307
Copy link
Contributor

@levy5307 levy5307 commented Mar 3, 2021

  1. fix heap-use-after-free in perf_counters
  2. Put the loading of configuration at the beginning of this func. Because in dsn_global_init(), it calls perf_counters::instance(), which calls shared_io_service::instance(). And in the cstor of shared_io_service, it calls dsn_config_get_value_uint64() to load the corresponding configs. That will make dsn_config_get_value_uint64() get wrong value if we put dsn_config_load in behind of dsn_global_init()

The error message asan reports:

==8642==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000000368 at pc 0x564e98eb17d9 bp 0x7ffe4593b030 sp 0x7ffe4593b020
READ of size 8 at 0x60b000000368 thread T0
    #0 0x564e98eb17d8 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::system::error_code&) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/deadline_timer_service.hpp:144
    #1 0x564e98eb17d8 in boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::cancel() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/basic_deadline_timer.hpp:310
    #2 0x564e98eb17d8 in dsn::perf_counter_number_percentile_atomic::~perf_counter_number_percentile_atomic() /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counter_atomic.h:222
    #3 0x564e98eb17d8 in dsn::perf_counter_number_percentile_atomic::~perf_counter_number_percentile_atomic() /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counter_atomic.h:222
    #4 0x564e98eaee71 in dsn::ref_counter::release_ref() /home/mi/workspace/pegasus/rdsn/include/dsn/utility/autoref_ptr.h:84
    #5 0x564e98eaee71 in dsn::ref_ptr<dsn::perf_counter>::~ref_ptr() /home/mi/workspace/pegasus/rdsn/include/dsn/utility/autoref_ptr.h:139
    #6 0x564e98eaee71 in dsn::perf_counters::counter_object::~counter_object() /home/mi/workspace/pegasus/rdsn/include/dsn/perf_counter/perf_counters.h:153
    #7 0x564e98eaee71 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>::~pair() /usr/include/c++/7/bits/stl_pair.h:208
    #8 0x564e98eaee71 in void __gnu_cxx::new_allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>*) /usr/include/c++/7/ext/new_allocator.h:140
    #9 0x564e98eaee71 in void std::allocator_traits<std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> >(std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> >&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>*) /usr/include/c++/7/bits/alloc_traits.h:487
    #10 0x564e98eaee71 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2084
    #11 0x564e98eaee71 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>, true> > >::_M_deallocate_nodes(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>, true>*) /usr/include/c++/7/bits/hashtable_policy.h:2097
    #12 0x564e98eaee71 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear() /usr/include/c++/7/bits/hashtable.h:2032
    #13 0x564e98e937a7 in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, dsn::perf_counters::counter_object, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, dsn::perf_counters::counter_object> > >::clear() /usr/include/c++/7/bits/unordered_map.h:842
    #14 0x564e98e937a7 in dsn::perf_counters::~perf_counters() /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counters.cpp:94
    #15 0x7f18a22530f0  (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)
    #16 0x7f18a22531e9 in exit (/lib/x86_64-linux-gnu/libc.so.6+0x431e9)
    #17 0x7f18a2231b9d in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b9d)
    #18 0x564e98be6119 in _start (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x27c119)

0x60b000000368 is located 104 bytes inside of 112-byte region [0x60b000000300,0x60b000000370)
freed by thread T0 here:
    #0 0x7f18a387a9c8 in operator delete(void*, unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe19c8)
    #1 0x564e98d57789 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::~deadline_timer_service() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/deadline_timer_service.hpp:79
    #2 0x564e98d65171 in boost::asio::detail::service_registry::destroy(boost::asio::execution_context::service*) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/service_registry.ipp:110
    #3 0x564e98d65171 in boost::asio::detail::service_registry::destroy_services() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/service_registry.ipp:54
    #4 0x564e98d65171 in boost::asio::execution_context::destroy() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/impl/execution_context.ipp:46
    #5 0x564e98d65171 in boost::asio::execution_context::~execution_context() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/impl/execution_context.ipp:35
    #6 0x564e98d65c23 in boost::asio::io_context::~io_context() /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/impl/io_context.ipp:55
    #7 0x564e98d65c23 in dsn::tools::shared_io_service::~shared_io_service() /home/mi/workspace/pegasus/rdsn/src/utils/shared_io_service.h:70
    #8 0x7f18a22530f0  (/lib/x86_64-linux-gnu/libc.so.6+0x430f0)

previously allocated by thread T0 here:
    #0 0x7f18a3879448 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0448)
    #1 0x564e98d6824a in boost::asio::execution_context::service* boost::asio::detail::service_registry::create<boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >, boost::asio::io_context>(void*) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/service_registry.hpp:87
    #2 0x564e98d64ae4 in boost::asio::detail::service_registry::do_use_service(boost::asio::execution_context::service::key const&, boost::asio::execution_context::service* (*)(void*), void*) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/service_registry.ipp:132
    #3 0x564e98e94c73 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >& boost::asio::detail::service_registry::use_service<boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> > >(boost::asio::io_context&) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/impl/service_registry.hpp:39
    #4 0x564e98e94c73 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >& boost::asio::use_service<boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> > >(boost::asio::io_context&) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/impl/io_context.hpp:39
    #5 0x564e98e94c73 in boost::asio::basic_io_object<boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >, true>::basic_io_object(boost::asio::io_context&) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/basic_io_object.hpp:224
    #6 0x564e98e94c73 in boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::basic_deadline_timer(boost::asio::io_context&) /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/basic_deadline_timer.hpp:159
    #7 0x564e98e94c73 in dsn::perf_counter_number_percentile_atomic::perf_counter_number_percentile_atomic(char const*, char const*, char const*, dsn_perf_counter_type_t, char const*) /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counter_atomic.h:215
    #8 0x564e98e94c73 in dsn::perf_counters::new_counter(char const*, char const*, char const*, dsn_perf_counter_type_t, char const*) /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counters.cpp:193
    #9 0x564e98e9b2c7 in dsn::perf_counters::get_global_counter(char const*, char const*, char const*, dsn_perf_counter_type_t, char const*, bool) /home/mi/workspace/pegasus/rdsn/src/perf_counter/perf_counters.cpp:126
    #10 0x564e98c1c0f4 in dsn::perf_counter_wrapper::init_global_counter(char const*, char const*, char const*, dsn_perf_counter_type_t, char const*) /home/mi/workspace/pegasus/rdsn/include/dsn/perf_counter/perf_counter_wrapper.h:99
    #11 0x564e98c1c0f4 in dsn::perf_counter_http_service_test_get_perf_counter_Test::TestBody() /home/mi/workspace/pegasus/rdsn/src/http/test/perf_counter_http_service_test.cpp:31
    #12 0x564e990c10a9 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x7570a9)
    #13 0x564e990bb3f6 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x7513f6)
    #14 0x564e9909f07b in testing::Test::Run() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x73507b)
    #15 0x564e9909f9a3 in testing::TestInfo::Run() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x7359a3)
    #16 0x564e990a001b in testing::TestCase::Run() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x73601b)
    #17 0x564e990a6ebf in testing::internal::UnitTestImpl::RunAllTests() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x73cebf)
    #18 0x564e990c21d0 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x7581d0)
    #19 0x564e990bc232 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x752232)
    #20 0x564e990a5aa5 in testing::UnitTest::Run() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x73baa5)
    #21 0x564e990cfb79 in RUN_ALL_TESTS() (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x765b79)
    #22 0x564e990cfb08 in main (/home/mi/workspace/pegasus/rdsn/builder/src/http/test/dsn_http_test+0x765b08)
    #23 0x7f18a2231b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-use-after-free /home/mi/workspace/pegasus/rdsn/thirdparty/output/include/boost/asio/detail/deadline_timer_service.hpp:144 in boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::cancel(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::system::error_code&)
Shadow bytes around the buggy address:
  0x0c167fff8010: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fd fd
  0x0c167fff8020: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c167fff8030: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c167fff8040: 00 00 fa fa fa fa fa fa fa fa 00 00 00 00 00 00
  0x0c167fff8050: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
=>0x0c167fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fa fa
  0x0c167fff8070: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x0c167fff8080: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
  0x0c167fff8090: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c167fff80a0: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c167fff80b0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==8642==ABORTING

@levy5307 levy5307 added the type/sanitize Fixes on errors reported by sanitizers. label Mar 3, 2021
@levy5307 levy5307 marked this pull request as draft March 4, 2021 02:30
@levy5307 levy5307 marked this pull request as ready for review March 4, 2021 02:45
@levy5307 levy5307 marked this pull request as draft March 4, 2021 03:08
@levy5307 levy5307 marked this pull request as ready for review March 4, 2021 05:38
@foreverneverer foreverneverer merged commit 6528eb6 into XiaoMi:master Mar 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type/sanitize Fixes on errors reported by sanitizers.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants