Fix numeric issues in PFCand scaling, add some debug output #41550

kdlong · 2023-05-05T07:58:26Z

PR description:

Fix candidate scaling in cases where momentum is a huge number, where the mass-aware scaling hits numeric issues. The fix is simple, just calculate E in a way without ratios involved. This should be a fix for #41397

PR validation:

Checked that the crash and nan candidate are fixed and the mass scaling works properly on the failing even from #41397

cmsbuild · 2023-05-05T08:08:41Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41550/35424

This PR adds an extra 56KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ParticleFlowCandidate/src/PFCandidate.cc modified in PR(s): PFCandidate no longer inherits from CompositeCandidate #38999
- File RecoParticleFlow/PFProducer/src/PFAlgo.cc modified in PR(s): [RECONSTRUCTION] [CLANG] Fix warnings reported by llvm16 in CLANG IBs #41543

cmsbuild · 2023-05-05T08:09:07Z

A new Pull Request was created by @kdlong (Kenneth Long) for master.

It involves the following packages:

DataFormats/ParticleFlowCandidate (reconstruction)
RecoParticleFlow/PFClusterTools (reconstruction)
RecoParticleFlow/PFProducer (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@mmarionncern, @rovere, @lgray, @missirol, @hatakeyamak, @seemasharmafnal this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2023-05-05T08:27:43Z

type pf

mmusich · 2023-05-05T08:27:49Z

type bug-fix

cmsbuild · 2023-05-05T08:58:57Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41550/35425

This PR adds an extra 56KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ParticleFlowCandidate/src/PFCandidate.cc modified in PR(s): PFCandidate no longer inherits from CompositeCandidate #38999
- File RecoParticleFlow/PFProducer/src/PFAlgo.cc modified in PR(s): [RECONSTRUCTION] [CLANG] Fix warnings reported by llvm16 in CLANG IBs #41543

cmsbuild · 2023-05-05T08:59:22Z

Pull request #41550 was updated. @cmsbuild, @mandrenguyen, @clacaputo can you please check and sign again.

perrotta · 2023-05-05T09:02:50Z

DataFormats/ParticleFlowCandidate/src/PFCandidate.cc

+
+  float e = std::sqrt(p() * p() * rescaleFactor * rescaleFactor + mass() * mass());
+
+  // Protect against invalid values (shouldn't happen, but could)


I wonder whether with the new formulation above (no more differences that could lead to numerical instabilities) this check is still needed. Did you try in your test if e can get some invalid value now?

No, I didn't see invalid values after this fix. I think what happened is that the scaling was applied multiple times, the first time being screwed up by the precision, and the second time giving a negative value for e^2 due to a effectively having a negative mass from the inconsistent E value. Probably this check isn't needed, or it could raise and exception instead of silently correcting the value. I'm open to suggestions.

mandrenguyen · 2023-05-07T09:15:44Z

I suggest we roll back my hotfix (##41473) in this PR, as this solution should address the problem at the source. Please also prepare a 13_1_X backport for consistency

kdlong · 2023-05-07T10:23:27Z

Thanks @mandrenguyen, I'll do this ASAP. Should I leave in the nan check in the rescaling function or drop it there as well?

mandrenguyen · 2023-05-07T10:40:50Z

I guess we should avoid checks that might end up obscuring problems.

cmsbuild · 2023-05-10T08:59:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41550/35500

This PR adds an extra 60KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/ParticleFlowCandidate/src/PFCandidate.cc modified in PR(s): PFCandidate no longer inherits from CompositeCandidate #38999

cmsbuild · 2023-05-10T09:00:08Z

Pull request #41550 was updated. @cmsbuild, @mandrenguyen, @clacaputo can you please check and sign again.

clacaputo · 2023-05-11T10:45:19Z

please test

cmsbuild · 2023-05-11T15:16:30Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2822e8/32558/summary.html
COMMIT: aef16e0
CMSSW: CMSSW_13_2_X_2023-05-11-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41550/32558/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

136.83111136.83111_RunJetHT2017FreMINIAODUL/step2_RunJetHT2017FreMINIAODUL.log
136.9136.9_RunDoubleMuon2016C/step2_RunDoubleMuon2016C.log
139.001139.001_RunMinimumBias2021/step2_RunMinimumBias2021.log

Expand to see more relval errors ...

Comparison Summary

Summary:

You potentially removed 3 lines from the logs
Reco comparison results: 19 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3461906
DQMHistoTests: Total failures: 35
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461849
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

clacaputo · 2023-05-15T08:11:38Z

please test

cmsbuild · 2023-05-15T10:50:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2822e8/32626/summary.html
COMMIT: aef16e0
CMSSW: CMSSW_13_2_X_2023-05-14-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41550/32626/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 22 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3461906
DQMHistoTests: Total failures: 44
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461840
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

mandrenguyen · 2023-05-15T19:38:38Z

+1

cmsbuild · 2023-05-15T19:39:04Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2023-05-16T05:14:58Z

+1

There are tiny differences in the masses of very few packedPFCandidates, which are apparently consistent with the changes at the level of numerical precision implemented with this fix

Fix numeric issues in PFCand scaling, add some debug output

98db177

cmsbuild added this to the CMSSW_13_2_X milestone May 5, 2023

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending code-checks-pending labels May 5, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels May 5, 2023

kdlong mentioned this pull request May 5, 2023

Fix pf candidate scaling #41551

Closed

cmsbuild added bug-fix pf labels May 5, 2023

Undo new line modification

4cf65aa

cmsbuild added code-checks-pending and removed code-checks-approved labels May 5, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels May 5, 2023

perrotta reviewed May 5, 2023

View reviewed changes

Remove nan check in pflinker

8b811d3

cmsbuild added code-checks-pending and removed code-checks-approved labels May 9, 2023

Remove nan check in pf cand rescaling

aef16e0

cmsbuild added code-checks-pending and removed code-checks-approved labels May 10, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels May 10, 2023

cmsbuild added tests-started and removed tests-pending labels May 11, 2023

cmsbuild added tests-rejected and removed tests-started labels May 11, 2023

cmsbuild added tests-started and removed tests-rejected labels May 15, 2023

cmsbuild added tests-approved and removed tests-started labels May 15, 2023

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels May 15, 2023

cmsbuild added orp-approved and removed orp-pending labels May 16, 2023

cmsbuild merged commit 8976b8a into cms-sw:master May 16, 2023

cmsbuild mentioned this pull request May 16, 2023

[ROOT628] Root update v6 28 00 patches cms-sw/cmsdist#8506

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix numeric issues in PFCand scaling, add some debug output #41550

Fix numeric issues in PFCand scaling, add some debug output #41550

kdlong commented May 5, 2023

cmsbuild commented May 5, 2023

cmsbuild commented May 5, 2023

mmusich commented May 5, 2023

mmusich commented May 5, 2023

cmsbuild commented May 5, 2023

cmsbuild commented May 5, 2023

perrotta May 5, 2023

kdlong May 5, 2023

mandrenguyen commented May 7, 2023

kdlong commented May 7, 2023

mandrenguyen commented May 7, 2023

cmsbuild commented May 10, 2023

cmsbuild commented May 10, 2023

clacaputo commented May 11, 2023

cmsbuild commented May 11, 2023

clacaputo commented May 15, 2023

cmsbuild commented May 15, 2023

mandrenguyen commented May 15, 2023

cmsbuild commented May 15, 2023

perrotta commented May 16, 2023


		float e = std::sqrt(p() * p() * rescaleFactor * rescaleFactor + mass() * mass());

		// Protect against invalid values (shouldn't happen, but could)

Fix numeric issues in PFCand scaling, add some debug output #41550

Fix numeric issues in PFCand scaling, add some debug output #41550

Conversation

kdlong commented May 5, 2023

PR description:

PR validation:

cmsbuild commented May 5, 2023

cmsbuild commented May 5, 2023

mmusich commented May 5, 2023

mmusich commented May 5, 2023

cmsbuild commented May 5, 2023

cmsbuild commented May 5, 2023

perrotta May 5, 2023

Choose a reason for hiding this comment

kdlong May 5, 2023

Choose a reason for hiding this comment

mandrenguyen commented May 7, 2023

kdlong commented May 7, 2023

mandrenguyen commented May 7, 2023

cmsbuild commented May 10, 2023

cmsbuild commented May 10, 2023

clacaputo commented May 11, 2023

cmsbuild commented May 11, 2023

RelVals-INPUT

Comparison Summary

clacaputo commented May 15, 2023

cmsbuild commented May 15, 2023

Comparison Summary

mandrenguyen commented May 15, 2023

cmsbuild commented May 15, 2023

perrotta commented May 16, 2023