Fix Phase2 ditau HLT path (with DeepTau Isolation)

[hsert]

PR description:

This PR implements the usage of Phase2 DeepTau training for ditau Phase 2 trigger, which was integrated with PR 42662. In the integrated version, Run2 training files were used. The problem with this implementation has been fixed and this PR proposes this fix and necessary changes.

Hopefully it will fix the issue mentioned here: #42862

The related JIRA ticket is CMSHLT-3010

PR validation:

The basic tests has been performed, suggested in the CMSSW PR instructions.

There was no failed tests.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43855/38694

• This PR adds an extra 60KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): [14_1_X] [Phase-2] Change Phase-2 HLT Egamma paths to use the GT emulator #43809

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43855/38694/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43855/38694/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[mmusich]

@hsert please apply the code-checks as suggested in the comment #43855 (comment) .

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43855/38706

• This PR adds an extra 56KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/python/HLT_75e33_cff.py modified in PR(s): [14_1_X] [Phase-2] Change Phase-2 HLT Egamma paths to use the GT emulator #43809

[cmsbuild]

A new Pull Request was created by @hsert (HaleSert) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)
• RecoTauTag/RecoTau (reconstruction)

@mandrenguyen, @mmusich, @Martin-Grunewald, @cmsbuild, @jfernan2 can you please review it and eventually sign? Thanks.
@rovere, @silviodonato, @missirol, @SohamBhattacharya, @mbluj, @Martin-Grunewald, @azotz this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03552d/37203/summary.html
COMMIT: ba5c296
CMSSW: CMSSW_14_0_X_2024-02-04-0000/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43855/37203/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 101 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 63 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3248614
• DQMHistoTests: Total failures: 1186
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247406
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[mmusich]

Tested positively running on lxplus921 (chosen in order to enable the AVX512F AVX512_VNNI instructions, see #42862 (comment)) preparing an area in the following way:

cd CMSSW_14_0_0_pre3/src/
cmsenv
git cms-addpkg HLTrigger/Configuration
git cms-addpkg RecoTauTag/RecoTau
git remote add hsert [email protected]:hsert/cmssw.git; git fetch hsert
git cherry-pick ba5c296e89dc1ab1dd1dfc8e3d4f233fff693518
scram b -j 20

then editing the file HLTrigger/Configuration/python/HLT_75e33_cff.py as follows:

diff --git a/HLTrigger/Configuration/python/HLT_75e33_cff.py b/HLTrigger/Configuration/python/HLT_75e33_cff.py
index e5ef776f070..83498d1ffa2 100644
--- a/HLTrigger/Configuration/python/HLT_75e33_cff.py
+++ b/HLTrigger/Configuration/python/HLT_75e33_cff.py
@@ -237,6 +237,8 @@ fragment.load("HLTrigger/Configuration/HLT_75e33/paths/HLTAnalyzerEndpath_cff")
 
 fragment.load('L1Trigger.Configuration.GTemulator_cff')
 fragment.l1tGTProducer.GMTTkMuons = cms.InputTag("l1tTkMuonsGmt")
+fragment.l1tGTProducer.CL2Jets = cms.InputTag("l1tSCPFL1PuppiCorrectedEmulator")
+fragment.l1tGTProducer.CL2HtSum = cms.InputTag("l1tSCPFL1PuppiCorrectedEmulatorMHT")
 fragment.GTemulatorTask = cms.Task(fragment.l1tGTProducer, fragment.l1tGTAlgoBlockProducer)
 fragment.GTemulation_step = cms.Path(cms.Sequence(fragment.GTemulatorTask))
 fragment.load('L1Trigger.Phase2L1GT.l1tGTMenu_cff')

(these two lines are needed, because there was some change in the L1 collections recently, and it doesn't work out of the box in the existing RelVals), then running

cmsDriver.py step3 -s HLT:@relval2026 --conditions auto:phase2_realistic_T25 --datatier GEN-SIM-DIGI-RAW -n -1 --eventcontent FEVTDEBUGHLT --geometry Extended2026D98 --era Phase2C17I13M9 --python step_3_cfg.py --no_exec --filein file:/eos/cms/store/group/offcomp_upgrade-sw/srimanob/phase2/HLT/Tau/step2.root --fileout file:step3.root --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000
cmsRun -e step_3_cfg.py

For the record I find in the FrameworkJobReport.xml

$ more FrameworkJobReport.xml | grep CPUModels
    <Metric Name="CPUModels" Value="Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz"/>

Without the commit of this PR (ba5c296e89dc1ab1dd1dfc8e3d4f233fff693518) the test above fails with:

2024-02-04 19:39:40.029941: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: Incompatible shapes: [0,1,1,64] vs. [154]
	 [[{{node inner_muon_norm_1/FusedBatchNorm_1/Mul}}]]
----- Begin Fatal Exception 04-Feb-2024 19:39:40 CET-----------------------
An exception of category 'InvalidRun' occurred while
   [0] Processing  Event run: 1 lumi: 999 event: 5 stream: 0
   [1] Running path 'HLT_DoubleMediumDeepTauPFTauHPS35_eta2p1'
   [2] Calling method for module DeepTauId/'hltHpsPFTauDeepTauProducer'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,64] vs. [154]
	 [[{{node inner_muon_norm_1/FusedBatchNorm_1/Mul}}]]
----- End Fatal Exception -------------------------------------------------
Another exception was caught while trying to clean up files after the primary fatal exception.

[mmusich]

+hlt

• for hsert@ba5c296
• see details of the cross-check at Fix Phase2 ditau HLT path (with DeepTau Isolation) #43855 (comment)
• changes detected at Fix Phase2 ditau HLT path (with DeepTau Isolation) #43855 (comment) are spurious and are coming from known workflow instabilities:
  • spurious differences in outputs of Run-3 wfs *.7 #39803
  • Non reproducibility in wf 136.793 #43293
  • Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999 #43790

[mmusich]

@hsert, would you mind editing the PR title to be more readable for release notes?

[hsert]

@hsert, would you mind editing the PR title to be more readable for release notes?

sure, is it better now?

[mmusich]

sure, is it better now?

yes, thank you.

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

Milestone for this pull request has been moved to CMSSW_14_1_X. Please open a backport if it should also go in to CMSSW_14_0_X.

[rappoccio]

+1

[mmusich]

@hsert, this update did not make it into CMSSW_14_0_0_pre3 (the release milestone was changed in the meanwhile). Can you create a backport for it?

[mmusich]

Can you create a backport for it?

to speed-up the process, I've taken the liberty of opening #43912.

[hsert]

Can you create a backport for it?

to speed-up the process, I've taken the liberty of opening #43912.

sorry for the delayed reply, and thanks for speeding up the process @mmusich.