Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASAN] heap-buffer-overflow in HLTLogMonitorFilter::filter #36729

Closed
iarspider opened this issue Jan 18, 2022 · 11 comments
Closed

[ASAN] heap-buffer-overflow in HLTLogMonitorFilter::filter #36729

iarspider opened this issue Jan 18, 2022 · 11 comments
Labels
fully-signed hlt-approved

Comments

@iarspider
Copy link
Contributor

Full log here

=================================================================
==21960==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020001b16f6 at pc 0x2b1b04a532a1 bp 0x7fff105710f0 sp 0x7fff105710e8
READ of size 1 at 0x6020001b16f6 thread T0
    #0 0x2b1b04a532a0 in HLTLogMonitorFilter::filter(edm::StreamID, edm::Event&, edm::EventSetup const&) const (.../pluginHLTriggerspecialAuto.so+0x38d2a0)
    #1 0x2b1add28b96b in edm::global::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (.../libFWCoreFramework.so+0x87296b)
    #2 0x2b1add27bae4 in edm::WorkerT<edm::global::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (.../libFWCoreFramework.so+0x862ae4)
    #3 0x2b1adcf73bc4 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (.../libFWCoreFramework.so+0x55abc4)
    #4 0x2b1adcf74733 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (.../libFWCoreFramework.so+0x55b733)
    #5 0x2b1adcf803f9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (.../libFWCoreFramework.so+0x5673f9)
    #6 0x2b1adcbeb1d1 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (.../libFWCoreFramework.so+0x1d21d1)
    #7 0x2b1adf257a58 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
    #8 0x2b1adf257a58 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
    #9 0x2b1adf257a58 in tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
    #10 0x2b1adccba85b in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (.../libFWCoreFramework.so+0x2a185b)
    #11 0x2b1adccec8a0 in edm::EventProcessor::runToCompletion() (.../libFWCoreFramework.so+0x2d38a0)
    #12 0x40e711 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (.../slc7_amd64_gcc11/cmsRun+0x40e711)
    #13 0x2b1adf245897 in tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/arena.cpp:698
    #14 0x411e7f in main::{lambda()#1}::operator()() const (.../slc7_amd64_gcc11/cmsRun+0x411e7f)
    #15 0x40c615 in main (.../slc7_amd64_gcc11/cmsRun+0x40c615)
    #16 0x2b1ae00c1554 in __libc_start_main (/lib64/libc.so.6+0x22554)
    #17 0x40c968  (.../slc7_amd64_gcc11/cmsRun+0x40c968)

0x6020001b16f6 is located 0 bytes to the right of 6-byte region [0x6020001b16f0,0x6020001b16f6)
allocated by thread T0 here:
    #0 0x2b1adc0f7d07 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b1b04a4f813 in HLTLogMonitorFilter::filter(edm::StreamID, edm::Event&, edm::EventSetup const&) const (.../pluginHLTriggerspecialAuto.so+0x389813)
    #2 0x2b1add28b96b in edm::global::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (.../libFWCoreFramework.so+0x87296b)
    #3 0x2b1add27bae4 in edm::WorkerT<edm::global::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (.../libFWCoreFramework.so+0x862ae4)
    #4 0x2b1adcf73bc4 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (.../libFWCoreFramework.so+0x55abc4)
    #5 0x2b1adcf74733 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (.../libFWCoreFramework.so+0x55b733)
    #6 0x2b1adcf803f9 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (.../libFWCoreFramework.so+0x5673f9)
    #7 0x2b1adcbeb1d1 in tbb::detail::d1::function_task<edm::WaitingTaskHolder::doneWaiting(std::__exception_ptr::exception_ptr)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (.../libFWCoreFramework.so+0x1d21d1)
    #8 0x2b1adf257a58 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
    #9 0x2b1adf257a58 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
    #10 0x2b1adf257a58 in tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/task_dispatcher.cpp:168
    #11 0x2b1adccba85b in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (.../libFWCoreFramework.so+0x2a185b)
    #12 0x2b1adccec8a0 in edm::EventProcessor::runToCompletion() (.../libFWCoreFramework.so+0x2d38a0)
    #13 0x40e711 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (.../slc7_amd64_gcc11/cmsRun+0x40e711)
    #14 0x2b1adf245897 in tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-e0f9a7ae89f8f641bc3af553000ba177/tbb-v2021.4.0/src/tbb/arena.cpp:698
    #15 0x411e7f in main::{lambda()#1}::operator()() const (.../slc7_amd64_gcc11/cmsRun+0x411e7f)
    #16 0x40c615 in main (.../slc7_amd64_gcc11/cmsRun+0x40c615)
    #17 0x2b1ae00c1554 in __libc_start_main (/lib64/libc.so.6+0x22554)

SUMMARY: AddressSanitizer: heap-buffer-overflow (.../pluginHLTriggerspecialAuto.so+0x38d2a0) in HLTLogMonitorFilter::filter(edm::StreamID, edm::Event&, edm::EventSetup const&) const
Shadow bytes around the buggy address:
  0x0c048002e280: fa fa fd fd fa fa fd fd fa fa fd fa fa fa fd fd
  0x0c048002e290: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fd
  0x0c048002e2a0: fa fa fd fa fa fa fd fd fa fa fd fd fa fa fd fa
  0x0c048002e2b0: fa fa fd fa fa fa 00 fa fa fa fd fd fa fa fd fd
  0x0c048002e2c0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fa
=>0x0c048002e2d0: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa[06]fa
  0x0c048002e2e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002e2f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002e300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002e310: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c048002e320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==21960==ABORTING
=== End log file ===
@cmsbuild
Copy link
Contributor

A new Issue was created by @iarspider .

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@iarspider
Copy link
Contributor Author

assign hlt

@cmsbuild
Copy link
Contributor

New categories assigned: hlt

@missirol,@Martin-Grunewald you have been requested to review this Pull request/Issue and eventually sign? Thanks

@Martin-Grunewald
Copy link
Contributor

@fwyzard @Dr15Jones
Chris, you modified this most recently...

@fwyzard
Copy link
Contributor

fwyzard commented Jan 18, 2022

  1. how does one reproduce this locally ?
  2. can we get line number information in the dumps, instead of just the binary addresses ?

@Dr15Jones
Copy link
Contributor

Dr15Jones commented Jan 18, 2022 •
edited
Loading

@fwyzard

how does one reproduce this locally ?

If you create a CMSSW_12_3_ASAN_X_2022-01-17-2300 work area you can run jobs under ASAN. Be sure to be on a machine that lets you have unlimited vmem (I can't do it at FNAL, only at CERN). As for which job, I can never figure out how to run the 'Other Tests'.

can we get line number information in the dumps, instead of just the binary addresses ?

If you checkout the code and rebuild with -g then ASAN will give line numbers.

@fwyzard
Copy link
Contributor

fwyzard commented Jan 18, 2022

thanks @Dr15Jones , I could rebuild and reproduce with

cmsrel CMSSW_12_3_ASAN_X_2022-01-17-2300
cd CMSSW_12_3_ASAN_X_2022-01-17-2300/src
cmsenv
git cms-addpkg HLTrigger/special
USER_CXXFLAGS="-g" scram b -j
cd HLTrigger/special/test/
scram b runtests
...
Package HLTrigger/special: Running test testHLTriggerspecialLogMonitorFilter
 
===== Test "testHLTriggerspecialLogMonitorFilter" ====
Failure using testLogMonitorFilter.py: status 1
=== Log file ===
...
=================================================================
==9442==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020001b4516 at pc 0x7f78ea20d2a1 bp 0x7ffd76dfad30 sp 0x7ffd76dfad28
READ of size 1 at 0x6020001b4516 thread T0
    #0 0x7f78ea20d2a0 in HLTLogMonitorFilter::filter(edm::StreamID, edm::Event&, edm::EventSetup const&) const /tmp/fwyzard/CMSSW_12_3_ASAN_X_2022-01-17-2300/src/HLTrigger/special/plugins/HLTLogMonitorFilter.cc:198
    #1 0x7f7912d2c96b in edm::global::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-01-17-2300/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x87296b)
...

0x6020001b4516 is located 0 bytes to the right of 6-byte region [0x6020001b4510,0x6020001b4516)
allocated by thread T0 here:
    #0 0x7f7912fccd07 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x7f78ea209813 in HLTLogMonitorFilter::filter(edm::StreamID, edm::Event&, edm::EventSetup const&) const /cvmfs/cms-ib.cern.ch/nweek-02716/slc7_amd64_gcc11/external/gcc/11.2.1-f478fee2760dbd22aaabb4e3a8fe1640/include/c++/11.2.1/ext/new_allocator.h:127
...

@fwyzard
Copy link
Contributor

fwyzard commented Jan 18, 2022

OK, #36738 should fix the error.

@iarspider
Copy link
Contributor Author

@fwyzard thanks!

@missirol missirol mentioned this issue Jan 19, 2022
Merged
@missirol
Copy link
Contributor

+hlt

@cmsbuild
Copy link
Contributor

This issue is fully signed and ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fully-signed hlt-approved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@iarspider @Dr15Jones @fwyzard @smuzaffar @cmsbuild @Martin-Grunewald @missirol