spurious differences in outputs of Run-3 wfs *.7

[missirol]

Differences in outputs of PR tests for wf 11634.7 were noticed in recent PRs to 12_5_X.

In each of these cases, (1) the PR was purely technical and almost-certainly incapable of creating changes to physics outputs, and (2) PR tests ran on IB CMSSW_12_5_X_2022-10-20-1100 (but I don't know if this type of issue had been seen before).

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1ff86b/28394/summary.html
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e7334d/28397/summary.html
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-107837/28401/summary.html

In one case (#39793 (comment)), PR tests were re-run (using the same IB as base), and after that bin-by-bin differences for wf 11634.7 disappeared, suggesting some non-reproducibility is at play.

The corresponding PRs to 12_4_X and 12_6_X (tested just as recently) didn't exhibit this issue.

Edit : originally, these spurious differences were only seen in 12_5_X; later on, they also appeared in the master branch (13_0_X at the time).

Edit (May 24th, 2023):

For the record, #41471 (and backports) removed wf 11634.7 (2022 HLT and MC GT) from the 'limited matrix' in CMSSW_13_X_Y, and effectively replaced it with wf 12434.7 (2023 HLT and MC GT).

Edit (March 20th, 2024): #39803 (comment)

[cmsbuild]

A new Issue was created by @missirol Marino Missiroli.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

assign reconstruction, tracking-pog

[mmusich]

11634.7 is a dedicated extended mkFit setup.

$ runTheMatrix.py -nel  11634.7 

11634.7 2021_trackingMkFit+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano [1]: cmsDriver.py TTbar_14TeV_TuneCP5_cfi  -s GEN,SIM -n 10 --conditions auto:phase1_2022_realistic --beamspot Realistic25ns13p6TeVEarly2022Collision --datatier GEN-SIM --eventcontent FEVTDEBUG --geometry DB:Extended --era Run3 --relval 9000,100 
                                           [2]: cmsDriver.py step2  -s DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2022 --conditions auto:phase1_2022_realistic --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry DB:Extended --era Run3 --customise RecoTracker/MkFit/customizeHLTIter0ToMkFit.customizeHLTIter0ToMkFit
                                           [3]: cmsDriver.py step3  -s RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,NANO,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM+@nanoAODDQM --conditions auto:phase1_2022_realistic --datatier GEN-SIM-RECO,MINIAODSIM,NANOAODSIM,DQMIO -n 10 --eventcontent RECOSIM,MINIAODSIM,NANOEDMAODSIM,DQM --geometry DB:Extended --era Run3 --procModifiers trackingMkFitDevel
                                           [4]: cmsDriver.py step4  -s HARVESTING:@standardValidation+@standardDQM+@ExtraHLT+@miniAODValidation+@miniAODDQM+@nanoAODDQM --conditions auto:phase1_2022_realistic --mc  --geometry DB:Extended --scenario pp --filetype DQM --era Run3 -n 100 

1 workflows with 4 steps

 -------------------------------------------------------------------------------- 

[missirol]

assign reconstruction, tracking-pog

[cmsbuild]

New categories assigned: tracking-pog,reconstruction

@slava77,@mmusich,@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

[missirol]

#39811 provides another example (again in 12_5_X):

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b4c6cc/28422/summary.html

[missirol]

(but I don't know if this type of issue had been seen before)

I checked recent 12_5_X PRs for which PR-tests are still accessible, and I didn't find other ones affected by this issue. So, I cannot exclude that this issue somehow started only since CMSSW_12_5_X_2022-10-20-1100.

[missirol]

#39814 provides another example, again in 12_5_X (enough examples at this point):

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b1c64e/28423/summary.html

[slava77]

#39811 provides another example (again in 12_5_X):

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b4c6cc/28422/summary.html

there is one more pixelPair step track candidate relative to the baseline https://tinyurl.com/26w5bw6a
This iteration is not using mkFit. So, it's not obvious why the difference would be localized in the mkfit wf.

do these diffs show up in 12_5_X only or also in 12_6_X ?

[slava77]

there is one more pixelPair step track candidate relative to the baseline https://tinyurl.com/26w5bw6a

and apparently 2 "existing" track candidates are different (in addition to having one more), based on e.g. chi2 distr

[slava77]

This iteration is not using mkFit. So, it's not obvious why the difference would be localized in the mkfit wf.

uhm, I'm wrong, pixelPair in this setup is using mkfit as well

[mmusich]

pixelPair in this setup is using mkfit as well

right

cmssw/Configuration/ProcessModifiers/python/trackingMkFitDevel_cff.py

Line 26 in bf6b0cc

trackingMkFitPixelPairStep,

do these diffs show up in 12_5_X only or also in 12_6_X ?

I am a bit surprised it doesn't show (at least there haven't been reports) in 12_6_X as well.

[perrotta]

urgent
(marking urgent the issues affecting relvals in the IBs)

[mmusich]

It looks like this is starting to hit also master,
see e.g.:

• Run3-gex145 Change getByLabel to use tokens in Alignment/OfflineValidation #40051 (comment)
• minor improvements to Alignment/OfflineValidation #40222 (comment)

[makortel]

Also here #40133 (comment)