Use likelihood fit instead of chi-square fit in DQMServices slice fits #43106

guitargeek · 2023-10-24T22:20:16Z

The DQM plots use the TH2::FitSlicesY() function to fit some Gaussians. However, some of the fits are failing. This was not resulting in errors so far, but with the switch to Minuit2 by default in ROOT 6.30 it will.

The problem is that it uses chi-square fits to fit slices with many empty bins, which is not appropriate. Doing a likelihood fit with the "l" option is one way to fix the problem, because it can better deal with empty bins.

Thanks to @lmoneta for this suggestion!
See root-project/root#13852

Closes #42979.

@smuzaffar, needs to be tested with ROOT master or 6.30 if possible.

The DQM plots use the `TH2::FitSlicesY()` function to fit some Gaussians. However, some of the fits are failing. This was not resulting in errors so far, but with the switch to Minuit2 by default in ROOT 6.30 it will. The problem is that it uses chi-square fits to fit slices with many empty bins, which is not appropriate. Doing a likelihood fit with the `"l"` option is one way to fix the problem, because it can better deal with empty bins. Closes #42979.

cmsbuild · 2023-10-24T22:28:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43106/37349

This PR adds an extra 24KB to repository

cmsbuild · 2023-10-24T22:28:35Z

A new Pull Request was created by @guitargeek (Jonas Rembser) for master.

It involves the following packages:

DQMServices/Components (dqm)

@tjavaid, @syuvivida, @rvenditti, @nothingface0, @cmsbuild, @antoniovagnerini can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

smuzaffar · 2023-10-25T12:29:50Z

test parameters:

workflow = 4.63,21.0,24.0,36.0,43.0,250409.0

smuzaffar · 2023-10-25T12:30:01Z

please test for CMSSW_13_3_ROOT630_X

smuzaffar · 2023-10-25T12:33:28Z

please test

cmsbuild · 2023-10-25T15:22:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3e8a50/35411/summary.html
COMMIT: dfced23
CMSSW: CMSSW_13_3_X_2023-10-24-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43106/35411/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 66 lines to the logs
Reco comparison results: 29 differences found in the comparisons
DQMHistoTests: Total files compared: 55
DQMHistoTests: Total histograms compared: 3789147
DQMHistoTests: Total failures: 61906
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3727209
DQMHistoTests: Total skipped: 32
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
Checked 236 log files, 187 edm output root files, 55 DQM output files
TriggerResults: no differences found

smuzaffar · 2023-10-25T15:32:41Z

please test

There are too many comparison differences, so lets re-run based on latest IB

guitargeek · 2023-10-25T15:42:07Z

I would not be surprised if there are real differences, given that the DQM Histograms are now fit without the chi-square approximation

cmsbuild · 2023-10-25T19:18:45Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3e8a50/35417/summary.html
COMMIT: dfced23
CMSSW: CMSSW_13_3_X_2023-10-25-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43106/35417/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 22 differences found in the comparisons
DQMHistoTests: Total files compared: 56
DQMHistoTests: Total histograms compared: 3795089
DQMHistoTests: Total failures: 61805
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3733252
DQMHistoTests: Total skipped: 32
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 55 files compared)
Checked 237 log files, 188 edm output root files, 56 DQM output files
TriggerResults: no differences found

mmusich · 2023-10-26T09:44:55Z

I would not be surprised if there are real differences, given that the DQM Histograms are now fit without the chi-square approximation

by naively looking into some of the histograms changed (e.g. https://tinyurl.com/yo6kah8c or https://tinyurl.com/ywu9hhm2) the fit quality looks better now.

cmsbuild · 2023-10-26T11:05:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3e8a50/35409/summary.html
COMMIT: dfced23
CMSSW: CMSSW_13_3_ROOT630_X_2023-10-24-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/43106/35409/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 61 lines to the logs
Reco comparison results: 35 differences found in the comparisons
DQMHistoTests: Total files compared: 55
DQMHistoTests: Total histograms compared: 3789147
DQMHistoTests: Total failures: 63259
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3725856
DQMHistoTests: Total skipped: 32
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
Checked 236 log files, 187 edm output root files, 55 DQM output files
TriggerResults: no differences found

smuzaffar · 2023-11-01T06:41:52Z

@cms-sw/dqm-l2 can you please review it. This fixes the Relvals failures for ROOT 6.28/30 IBs

smuzaffar · 2023-11-01T08:38:53Z

@rappoccio @antoniovilela can we merge this for 11h00 IB so thatit can go in ROOT 6.28/30 builds too

antoniovilela · 2023-11-01T14:06:17Z

+1

antoniovilela · 2023-11-01T14:08:17Z

@rappoccio @antoniovilela can we merge this for 11h00 IB so thatit can go in ROOT 6.28/30 builds too

@smuzaffar
Hi Shahzad,
Try to ping us using @cms-sw/orp-l2. I seem to pick it up faster.
Thanks,

AdrianoDee · 2024-01-09T16:14:07Z

For the records this PR is the cause of some failures in the 13_3_0_pre5 given the worsening of track/muons resolutions. E.g.:

guitargeek · 2024-01-09T16:34:50Z

Can you maybe post some plots here for non-CMS members like me? 🙂

davidlange6 · 2024-01-09T16:53:44Z

one example is this one - the fit is doing what it should (red chi2, green likelihood) - however, I suspect the intent is to estimated a resolution while ignoring the tails (which is not what the code does..)

guitargeek · 2024-01-09T16:59:27Z

Thanks David! Indeed neither fit looks appropriate here, maybe better just take the std. dev from the histogram or as as you said fit only the central region.

AdrianoDee · 2024-01-09T17:28:20Z

Yes, it's more a problem related to the way we calculate the resolution itself rather than directly originating from this PR (that basically highlights the issue).

Still one could argue that red is still better than green since the likelihood gives more importance to the tails, in the specific case posted here and elsewhere. Therefore the failure reports.

guitargeek · 2024-01-09T17:36:37Z

Relying on accidental properties of the chi2 fit is quite a random way to discount the tails. @AdrianoDee, what to you thing about Davids suggestion to only fit the central region (which could then be done with either chi2 or likelihood)?

mmusich · 2024-01-09T18:23:18Z

what to you thing about Davids suggestion to only fit the central region (which could then be done with either chi2 or likelihood)?

chiming in my (unrequested) 2 cents:

one of the points that we should not forget is that DQMGenericClient (as the name sorts of betrays) was originally devised as a generic tool to perform gaussian fits in a variety of DQM harvesting applications. Thanks to this PR we have now (re-?) discovered that this tool is used sometimes inappropriately to fit (also) non-gaussian distributions (e.g. residuals). On the other hand one might assume that the bulk of the use cases is using it appropriately (e.g. pulls), so restricting the fit to the central region might be erring on the opposite side.
all in all it is perhaps better to let to each one of the clients to choose in which modality to perform the fit (full range, limited range, not fitting at all, etc.) ?

makortel · 2024-01-16T15:56:56Z

How should we proceed here?

Does the <= 13_3_0_pre4 behavior need to be restored by 14_0_0_pre3 (last open prerelease)?

If yes, are the improvements in DQMGenericClient (along what @mmusich outlined in #43106 (comment)) something that could be developed, tested, and deployed in that time frame?

I guess, in principle, a possible "quick fix" would be to revert to ROOT 6.26 and the chi2 based fitting (while the improvements are being worked on), but I see that only as the last resort.

mmusich · 2024-01-16T16:35:53Z

if yes, are the improvements in DQMGenericClient (along what @mmusich outlined in #43106 (comment)) something that could be developed, tested, and deployed in that time frame?

there seems to be already something that does limited range fits:

cmssw/DQMServices/Components/plugins/DQMGenericClient.cc

Line 934 in 10b8a60

limitedFit(srcME, meanME, sigmaME);

perhaps it's just a matter of adjusting configurations. @cms-sw/tracking-pog-l2 @cms-sw/muon-pog-l2 for your consideration.

smuzaffar · 2024-01-16T17:30:10Z

@guitargeek , during offline release planning meeting today, we discussed if it is possible to use the old Minuit (instead of new default Minit2 ). Do you know if it is possible and how?

guitargeek · 2024-01-16T17:34:25Z

In which scope? All of CMSSW?

smuzaffar · 2024-01-16T17:39:29Z

yes all cmssw ( e.g. may be build root to use old Minuit as default)

makortel · 2024-01-16T18:59:09Z

From root-project/root#13661 and root-project/root#13852 it seem like

ROOT::Math::MinimizerOptions::SetDefaultMinimizer("Minuit");
// or
ROOT::Math::MinimizerOptions::SetDefaultMinimizer("Minuit", "Migrad");

would do the job. I suppose this function is thread-unsafe, so maybe the InitRootHandlers constructor would be a reasonable place to call it (especially given that this should be only a temporary workaround).

Then in the meanwhile @cms-sw/dqm-l2 @cms-sw/tracking-pog-l2 @cms-sw/muon-pog-l2 could look into restricting the fit ranges along #43106 (comment).

After that we could try again to move the DQMGenericClient (and others) to likelihood fits, after which we could move to Minuit2 by default (and all this while staying in Root 6.30). And probably now the discussion should be moved to a new issue.

makortel · 2024-01-16T19:03:24Z

I suppose this function is thread-unsafe

I found these

cmssw/DQM/SiStripCommissioningAnalysis/src/CalibrationAlgorithm.cc

Lines 88 to 90 in 10b8a60

    
           void CalibrationAlgorithm::analyse() { 
        
             ROOT::Math::MinimizerOptions::SetDefaultMinimizer("Minuit2", "Migrad"); 
        
             ROOT::Math::MinimizerOptions::SetDefaultStrategy(0);

cmssw/DQM/SiStripCommissioningAnalysis/src/CalibrationScanAlgorithm.cc

Lines 82 to 84 in 10b8a60

    
           void CalibrationScanAlgorithm::analyse() { 
        
             ROOT::Math::MinimizerOptions::SetDefaultMinimizer("Minuit2", "Migrad"); 
        
             ROOT::Math::MinimizerOptions::SetDefaultStrategy(0);

I guess (hope) these components are not run in standard workflows.

mmusich · 2024-01-16T19:12:03Z

I guess (hope) these components are not run in standard workflows.

IIRC, indeed they aren't. @rgerosa might confirm

rgerosa · 2024-01-16T19:21:48Z

Hi @mmusich yes I confirm these are just run in the analysis of tracker local-runs of type calibration scan

makortel · 2024-01-16T20:05:32Z

And probably now the discussion should be moved to a new issue.

The issue is here #43722 . I suggest to move all subsequent discussion on the topic there.

cmsbuild added this to the CMSSW_13_3_X milestone Oct 24, 2023

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Oct 24, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 24, 2023

cmsbuild added tests-started and removed tests-pending labels Oct 25, 2023

cmsbuild added tests-approved and removed tests-started labels Oct 25, 2023

cmsbuild added tests-started and removed tests-approved labels Oct 25, 2023

cmsbuild added tests-approved and removed tests-started labels Oct 25, 2023

smuzaffar mentioned this pull request Oct 26, 2023

CMSSW tests fails with Fatal Root Error: @SUB=Minuit2 #42979

Closed

cmsbuild added orp-approved and removed orp-pending labels Nov 1, 2023

cmsbuild merged commit da3c6d2 into cms-sw:master Nov 1, 2023
18 checks passed

guitargeek deleted the issue-fitslicesy branch November 1, 2023 17:20

smuzaffar mentioned this pull request Nov 2, 2023

Use likelihood fit instead of chi-square fit for PFJetDQMPostProcessor plugin #43170

Merged

smuzaffar mentioned this pull request Dec 4, 2023

Use likelihood fit instead of chi-square fit in DQMGenericClient::limitedFit #43490

Merged

aandvalenzuela mentioned this pull request Dec 15, 2023

CMSSW tests failing again with Fatal Root Error: @SUB=Minuit2 #43577

Closed

smuzaffar mentioned this pull request Dec 17, 2023

Use likelihood fit instead of chi-square fit in PVValidation #43588

Merged

makortel mentioned this pull request Jan 16, 2024

Deployment of likelihood fits in DQMGenericClient etc #43722

Open

smuzaffar mentioned this pull request Jan 24, 2024

Revert back to chi-square fit instead of likelihood fit #43782

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use likelihood fit instead of chi-square fit in DQMServices slice fits #43106

Use likelihood fit instead of chi-square fit in DQMServices slice fits #43106

guitargeek commented Oct 24, 2023 •

edited

Loading

cmsbuild commented Oct 24, 2023

cmsbuild commented Oct 24, 2023 •

edited

Loading

smuzaffar commented Oct 25, 2023

smuzaffar commented Oct 25, 2023

smuzaffar commented Oct 25, 2023

cmsbuild commented Oct 25, 2023

smuzaffar commented Oct 25, 2023

guitargeek commented Oct 25, 2023

cmsbuild commented Oct 25, 2023

mmusich commented Oct 26, 2023

cmsbuild commented Oct 26, 2023

smuzaffar commented Nov 1, 2023

smuzaffar commented Nov 1, 2023

antoniovilela commented Nov 1, 2023

antoniovilela commented Nov 1, 2023

AdrianoDee commented Jan 9, 2024

guitargeek commented Jan 9, 2024

davidlange6 commented Jan 9, 2024

guitargeek commented Jan 9, 2024

AdrianoDee commented Jan 9, 2024

guitargeek commented Jan 9, 2024

mmusich commented Jan 9, 2024

makortel commented Jan 16, 2024

mmusich commented Jan 16, 2024 •

edited

Loading

smuzaffar commented Jan 16, 2024

guitargeek commented Jan 16, 2024

smuzaffar commented Jan 16, 2024

makortel commented Jan 16, 2024

makortel commented Jan 16, 2024

mmusich commented Jan 16, 2024

rgerosa commented Jan 16, 2024

makortel commented Jan 16, 2024

Use likelihood fit instead of chi-square fit in DQMServices slice fits #43106

Use likelihood fit instead of chi-square fit in DQMServices slice fits #43106

Conversation

guitargeek commented Oct 24, 2023 • edited Loading

cmsbuild commented Oct 24, 2023

cmsbuild commented Oct 24, 2023 • edited Loading

smuzaffar commented Oct 25, 2023

smuzaffar commented Oct 25, 2023

smuzaffar commented Oct 25, 2023

cmsbuild commented Oct 25, 2023

Comparison Summary

smuzaffar commented Oct 25, 2023

guitargeek commented Oct 25, 2023

cmsbuild commented Oct 25, 2023

Comparison Summary

mmusich commented Oct 26, 2023

cmsbuild commented Oct 26, 2023

Comparison Summary

smuzaffar commented Nov 1, 2023

smuzaffar commented Nov 1, 2023

antoniovilela commented Nov 1, 2023

antoniovilela commented Nov 1, 2023

AdrianoDee commented Jan 9, 2024

guitargeek commented Jan 9, 2024

davidlange6 commented Jan 9, 2024

guitargeek commented Jan 9, 2024

AdrianoDee commented Jan 9, 2024

guitargeek commented Jan 9, 2024

mmusich commented Jan 9, 2024

makortel commented Jan 16, 2024

mmusich commented Jan 16, 2024 • edited Loading

smuzaffar commented Jan 16, 2024

guitargeek commented Jan 16, 2024

smuzaffar commented Jan 16, 2024

makortel commented Jan 16, 2024

makortel commented Jan 16, 2024

mmusich commented Jan 16, 2024

rgerosa commented Jan 16, 2024

makortel commented Jan 16, 2024

guitargeek commented Oct 24, 2023 •

edited

Loading

cmsbuild commented Oct 24, 2023 •

edited

Loading

mmusich commented Jan 16, 2024 •

edited

Loading