Index error in L1Trigger/L1TTrackMatch/L1TrackJetEmulatorProducer #43723

dan131riley · 2024-01-16T20:59:18Z

this line

cmssw/L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc

Line 298 in 10b8a60

    
           int j = eta_bin_firmwareStyle(L1TrkPtrs_[k]->getTanlWord());  //Function defined in L1TrackJetClustering.h

is occasionally producing a -1 bin index which is subsequently used to index into the stack-allocated epbins array, smashing the stack of the previous stack allocations. On very rare occasions this results in a segfault, but it actually happens (without a segfault) fairly often for some workflows. The way to see this to add

      assert(i < phiBins_ && i >= 0 && j < etaBins_ && j >= 0);

at line 300, and run wf 23234.0, 24834.0, or 25034.999, which should frequently give an assertion failure in step2 or step3. ASAN and UBSAN don't catch this, but valgrind memcheck does:

==113056== Thread 9:
==113056== Conditional jump or move depends on uninitialised value(s)
==113056==    at 0x1503E3C41: ap_fixed_base<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0>& ap_fixed_base<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0>::operator=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:665)
==113056==    by 0x15037286B: ap_fixed_base<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0>::ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:396)
==113056==    by 0x1503DCB9B: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::RType<14, 9, false>::plus ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) const (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1163)
==113056==    by 0x1503D77A3: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056==    by 0x1503D4203: L1TrackJetEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc:301)
==113056==  Uninitialised value was created by a stack allocation
==113056==    at 0x1503D777C: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056== 
==113056== Conditional jump or move depends on uninitialised value(s)
==113056==    at 0x1503DCCA3: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator=<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0>(ap_fixed_base<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:665)
==113056==    by 0x1503D77B6: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056==    by 0x1503D4203: L1TrackJetEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc:301)
==113056==  Uninitialised value was created by a st ack allocation
==113056==    at 0x1503D777C: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056== 
==113056== Conditional jump or move depends on uninitialised value(s)
==113056==    at 0x1503DCD86: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator=<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0>(ap_fixed_base<15, 10, false, (ap_q_mode)5, (ap_o_mode)3, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:745)
==113056==    by 0x1503D77B6: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056==    by 0x1503D4203: L1TrackJetEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc:301)
==113056==  Uninitialised value was created by a stack allocation
==113056==    at 0x1503D777C: ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>& ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>::operator+=<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0>(ap_fixed_base<14, 9, false, (ap_q_mode)5, (ap_o_mode)0, 0> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_fixed_base.h:1183)
==113056== 
==113056== Invalid read of size 1
==113056==    at 0x1503E0F13: ap_private<5, false, true>& ap_private<5, false, true>::operator+=<1, false>(ap_private<1, false, (1)<=(64)> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/etc/ap_private.h:1904)
==113056==    by 0x1503DD0EA: ap_int_base<5, false>& ap_int_base<5, false>::operator+=<1, false>(ap_int_base<1, false> const&) (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_int_base.h:667)
==113056==    by 0x1503D7964: ap_int_base<5, false>::operator++() (/cvmfs/cms.cern.ch/el9_amd64_gcc12/external/hls/2019.08-8afb4083e7b06154cf0bca6d787b688f/include/ap_int_base.h:693)
==113056==    by 0x1503D445A: L1TrackJetEmulatorProducer::produce(edm::Event&, edm::EventSetup const&) (L1Trigger/L1TTrackMatch/plugins/L1TrackJetEmulatorProducer.cc:313)

In all the cases I've checked, L1TrkPtrs_[k]->getTanlWord() at the failure has the value

$1 = {
  <ap_int_base<16, false>> = {
    <ssdm_int_sim<16, false>> = {
      V = {
        static mask = 65535,
        VAL = 45824
      }
    }, <No data fields>}, <No data fields>}

and working through eta_bin_firmwareStyle() confirms this gives a -1 index.

The text was updated successfully, but these errors were encountered:

cmsbuild · 2024-01-16T20:59:39Z

cms-bot internal usage

cmsbuild · 2024-01-16T20:59:40Z

A new Issue was created by @dan131riley Dan Riley.

@antoniovilela, @makortel, @smuzaffar, @Dr15Jones, @sextonkennedy, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2024-01-16T21:01:44Z

assign l1

cmsbuild · 2024-01-16T21:02:02Z

New categories assigned: l1

@epalencia,@aloeliger you have been requested to review this Pull request/Issue and eventually sign? Thanks

srimanob · 2024-01-29T02:25:14Z

Just a note from #43735 (closed, and follow up in this issue) is that it is unclear to me why the rates of failure are different with very close geometry version. For example, I try 20 jobs of ttbar nopu, 500 events per job with same GEN event, between 24834.0 and 25234.0. I don't see any fail job from 24834.0 (D98), but 4 job fail from 25234.0 (D99). However, it should be fixed before the Phase-2 production starts.

srimanob · 2024-01-29T02:26:02Z

@epalencia @aloeliger @BenjaminRS
Anything on L1T side to fix this issue? Thanks very much.

aloeliger · 2024-01-29T09:33:16Z

@BenjaminRS Do you know the track group responsible for this producer?

BenjaminRS · 2024-01-29T16:10:43Z

@SClarkPhysics - we have an issue with the line mentioned above. I see you added this particular line in this commit. Can you have a look into this problem as soon as you can please?
Thanks,
Benjamin

srimanob · 2024-02-01T07:18:49Z

To reproduce the error with CMSSW_14_0_0_pre2:

cmsDriver.py step2 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@relval2026 --conditions auto:phase2_realistic_T25 --datatier GEN-SIM-DIGI-RAW -n -1 --eventcontent FEVTDEBUGHLT --geometry Extended2026D98 --era Phase2C17I13M9 --python step2_D98_DIGIHLT_test.py --no_exec --filein file:step1_D98.root --fileout file:step2_D98.root --nThreads 1 --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000 --customise_commands "process.source.firstEvent = cms.untracked.uint32(200)"

with input file: /eos/cms/store/group/offcomp_upgrade-sw/srimanob/D99/HGCAL-DEBUG-3/GENSIM_D98_20072.root

I see that crash at event 209.

skinnari · 2024-02-01T14:48:45Z

Can maybe @NJManganelli help look at this from the GTT side?

ccahoughton · 2024-02-02T17:18:48Z

We are working on a fix. Should this be pushed to master? @srimanob

srimanob · 2024-02-02T17:28:12Z

We are working on a fix. Should this be pushed to master? @srimanob

Hi @ccahoughton, please. Thanks very much.

srimanob · 2024-02-03T14:18:16Z

Fix is in #43852

Testing in private production, issue is gone. I can completely produce sample with no crash (using 1000 events/lumi).

smuzaffar · 2024-10-16T09:34:54Z

Looks like this was fixed by #43852 .... @dan131riley can we close this issue?

dan131riley · 2024-10-16T11:39:04Z

Looks like this was fixed by #43852 .... @dan131riley can we close this issue?

ok to close

cmsbuild added the pending-assignment label Jan 16, 2024

cmsbuild added l1-pending pending-signatures and removed pending-assignment labels Jan 16, 2024

mmusich mentioned this issue Jan 17, 2024

Crash in Phase-2 L1TrackJetEmulatorProducer:l1tTrackJetsEmulation #43735

Closed

ccahoughton mentioned this issue Feb 2, 2024

Implement quick fix for eta going out of scope #43852

Merged

dan131riley mentioned this issue Feb 9, 2024

Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999 #43790

Open

dan131riley closed this as completed Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index error in L1Trigger/L1TTrackMatch/L1TrackJetEmulatorProducer #43723

Index error in L1Trigger/L1TTrackMatch/L1TrackJetEmulatorProducer #43723

dan131riley commented Jan 16, 2024

cmsbuild commented Jan 16, 2024 •

edited

Loading

cmsbuild commented Jan 16, 2024

makortel commented Jan 16, 2024

cmsbuild commented Jan 16, 2024

srimanob commented Jan 29, 2024

srimanob commented Jan 29, 2024

aloeliger commented Jan 29, 2024

BenjaminRS commented Jan 29, 2024

srimanob commented Feb 1, 2024 •

edited

Loading

skinnari commented Feb 1, 2024

ccahoughton commented Feb 2, 2024

srimanob commented Feb 2, 2024

srimanob commented Feb 3, 2024

smuzaffar commented Oct 16, 2024

dan131riley commented Oct 16, 2024

Index error in L1Trigger/L1TTrackMatch/L1TrackJetEmulatorProducer #43723

Index error in L1Trigger/L1TTrackMatch/L1TrackJetEmulatorProducer #43723

Comments

dan131riley commented Jan 16, 2024

cmsbuild commented Jan 16, 2024 • edited Loading

cmsbuild commented Jan 16, 2024

makortel commented Jan 16, 2024

cmsbuild commented Jan 16, 2024

srimanob commented Jan 29, 2024

srimanob commented Jan 29, 2024

aloeliger commented Jan 29, 2024

BenjaminRS commented Jan 29, 2024

srimanob commented Feb 1, 2024 • edited Loading

skinnari commented Feb 1, 2024

ccahoughton commented Feb 2, 2024

srimanob commented Feb 2, 2024

srimanob commented Feb 3, 2024

smuzaffar commented Oct 16, 2024

dan131riley commented Oct 16, 2024

cmsbuild commented Jan 16, 2024 •

edited

Loading

srimanob commented Feb 1, 2024 •

edited

Loading