Add prototype HcalNanoAOD #40489

DryRun · 2023-01-11T22:53:51Z

PR description:

Prototype implementation for use by HCAL DPG, not ready for merging yet

This PR implements a NanoAOD format for HCAL digis and RecHits. This data format is intended to replace the old HcalTupleMaker used by most HCAL workflows.

Example:

cmsDriver.py NANO \
    -s RAW2DIGI,RECO,USER:DPGAnalysis/HcalNanoAOD/hcalNano_cff.hcalNanoTask \
    --datatier NANOAOD \
    --eventcontent NANOAOD \
    --filein root://eoscms.cern.ch//store/group/dpg_hcal/comm_hcal/RAW/ZeroBias/361579/be9a0a12-548c-4aa0-837a-2d5654f8e63b.root \
    --fileout hcalnano_testcmsdriver.root \
    -n 100 \
    --nThreads 4 \
    --conditions auto:run3_data_prompt \
    --era Run3 \
    --python_filename testcmsdriver_cfg.py \
    --no_exec
cmsRun testcmsdriver_cfg.py

RecHits are implemented following @kdlong's PFNano branch, https://github.com/kdlong/cmssw/tree/pfNano_HCALOnly_12_6_0_pre4. The specific modules, however, are defined in DPGAnalysis/HcalNanoAOD (e.g. HBHERecHitFlatTableProducer), rather than DPGAnalysis/CaloNanoAOD (e.g. SimpleCaloRecHitFlatTableProducer). Hence, this package is independent of DPGAnalysis/CaloNanoAOD.

Digis are handled by dedicated modules in DPGAnalysis/HcalNanoAOD that output dense arrays, sorted by DetId. A list of the DetIds (i.e. the column labels) is placed in Run. For the moment, DetIds are also saved in Event, for development purposes although we might decide to keep them for ease of analysis.

Metadata from the uMNio, namely the labeling of calibration events (pedestal/LED/laser) and which type of laser event, is also handled by a dedicated module.

Two customization functions are provided for special datasets:

hcalNano_cff.customiseHcalCalib: for calibration sequence events in the TestEnablesEcalHcal dataset (i.e. events in the abort gap of type pedestal, LED, or laser). Specifies the correct label for the raw data; adds uMNio metadata; adds an important HLT filter to all Paths and Endpaths, which selects events triggered by HLT_HcalCalibration. This filter essential to avoid a crash, because ECAL-triggered events are missing the HCAL raw data collection (aside: HCAL needs to revisit the definition of this dataset; the trigger is based on a majority-wins vote of calibration flags from the HCAL uHTRs. A single uHTR flagging the event as calibration should be enough to consider the event calibration and not real physics!).
hcalNano_cff.customiseHcalLocal: for HCAL local runs. Loads the raw data using HcalTBSource; adjusts the number of time slices per digi; adds uMNio metadata.

TBD: trigger primitives.

PR validation:

Produces useful HCAL NanoAOD ntuples. Data validated event-by-event against HcalTupleMaker. Calibration and local run customization functions work on recent MWGR 2023 data.

Other

The contents of this PR are ready for use by HCAL DPG right away. An eventual merge should take into account @kdlong's PFNanoAOD branch, but this package is independent of DPGAnalysis/CaloNanoAOD.

@mariadalfonso @wang-hui

cmsbuild · 2023-01-11T23:00:39Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33666

This PR adds an extra 92KB to repository
Found files with invalid states:
- DPGAnalysis/HcalNanoAOD/test/testrechits_cfg.py:
  - Added: 03267a5
  - Deleted: afda26e
- DPGAnalysis/HcalNanoAOD/python/hcalnano_cff.py:
  - Added: 03267a5
  - Deleted: 3335021
- DPGAnalysis/HcalNanoAOD/test/testdigis_cfg.py:
  - Added: 03267a5
  - Deleted: afda26e

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33666/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33666/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33666/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33666/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2023-01-18T22:13:40Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33793

This PR adds an extra 116KB to repository
Found files with invalid states:
- DPGAnalysis/HcalNanoAOD/test/testrechits_cfg.py:
  - Added: 03267a5
  - Deleted: afda26e
- DPGAnalysis/HcalNanoAOD/python/hcalSortedDigiTable_cff.py:
  - Added: 03267a5
  - Deleted: 830caba
- DPGAnalysis/HcalNanoAOD/python/hcalnano_cff.py:
  - Added: 03267a5
  - Deleted: 3335021
- DPGAnalysis/HcalNanoAOD/test/testdigis_cfg.py:
  - Added: 03267a5
  - Deleted: afda26e
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): avoid use of auto:run*_data_relval GTs in non-reHLT wfs #40481

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33793/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33793/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly

cmsbuild · 2023-01-19T22:47:26Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/33814

This PR adds an extra 24KB to repository

cmsbuild · 2023-01-19T22:47:52Z

A new Pull Request was created by @DryRun (David Yu) for master.

It involves the following packages:

DPGAnalysis/HcalNanoAOD (****)
DataFormats/HcalDetId (simulation)

The following packages do not have a category, yet:

DPGAnalysis/HcalNanoAOD
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@cmsbuild, @civanch, @mdhildreth can you please review it and eventually sign? Thanks.
@missirol, @mariadalfonso, @abdoulline, @rovere, @bsunanda this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mariadalfonso · 2023-01-25T13:24:34Z

@vlimant @swertz
this should be assigned to XPOG

Trk and ECAL have already their nano merged, while HCAL and muon'PR (#38226) are made and waiting your review

perrotta · 2023-01-25T13:27:23Z

assign xpog

perrotta · 2023-01-25T13:27:38Z

enable nano

cmsbuild · 2023-01-25T13:27:47Z

New categories assigned: xpog

@swertz,@vlimant you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild · 2023-02-20T01:48:31Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e121db/30738/summary.html
COMMIT: 1484144
CMSSW: CMSSW_13_1_X_2023-02-19-0000/el8_amd64_gcc11
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40489/30738/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3529029
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3529007
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 11
DQMHistoTests: Total histograms compared: 10829
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 10829
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 10 files compared)
Checked 23 log files, 10 edm output root files, 11 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.232	2.232	0.000 ( +0.0% )	9.54	9.48	+0.6%	1.517	1.467
2500.311	2.323	2.323	0.000 ( +0.0% )	9.25	9.06	+2.1%	1.890	1.833
2500.312	2.277	2.277	0.000 ( +0.0% )	9.30	9.36	-0.6%	1.869	1.843
2500.33	1.100	1.100	0.000 ( +0.0% )	22.07	21.10	+4.6%	1.647	1.644
2500.331	1.394	1.394	0.000 ( +0.0% )	16.17	15.92	+1.6%	1.796	1.797
2500.332	1.326	1.326	0.000 ( +0.0% )	18.03	17.77	+1.4%	1.861	1.861
2500.401	2.139	2.139	0.000 ( +0.0% )	10.45	10.40	+0.4%	1.198	1.153
2500.501	1.711	1.711	0.000 ( +0.0% )	16.86	16.57	+1.7%	1.116	1.088
2500.511	1.124	1.124	0.000 ( +0.0% )	30.99	30.61	+1.3%	1.370	1.324
2500.601	2.050	2.050	0.000 ( +0.0% )	12.57	12.49	+0.6%	1.176	1.137

vlimant · 2023-02-20T12:37:12Z

@DryRun please amend the PR description accordingly

cmsbuild · 2023-03-07T22:09:48Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34495

This PR adds an extra 16KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34495/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34495/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34495/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34495/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2023-03-07T23:11:27Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40489/34496

This PR adds an extra 28KB to repository

cmsbuild · 2023-03-07T23:11:50Z

Pull request #40489 was updated. @cmsbuild, @civanch, @swertz, @mdhildreth, @vlimant can you please check and sign again.

DryRun · 2023-03-07T23:15:48Z

@DryRun please amend the PR description accordingly

Done. I made an additional commit to support running over HCAL local runs (hopefully the last feature modification in this PR, we should put additional features into the next PR).

vlimant · 2023-04-03T12:35:41Z

can you please add a test workflow in the matrix, like was done for #41245, so that we keep a eye on this alternative nano configuration

DryRun · 2023-05-10T05:09:02Z

@jhakala added a test workflow. I rebased the PR to 13_1_0_pre4 and renamed the branch accordingly, but seem to have caused this PR to close permanently, similar to isaacs/github#361. I think it's easiest to make a new PR (#41607), but please advise if I should try to recover this one.

cmsbuild added this to the CMSSW_13_0_X milestone Jan 11, 2023

cmsbuild added code-checks-pending new-package-pending operations-pending orp-pending pending-signatures simulation-pending tests-pending labels Jan 11, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 11, 2023

cmsbuild mentioned this pull request Jan 13, 2023

avoid use of auto:run*_data_relval GTs in non-reHLT wfs #40481

Merged

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 18, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 18, 2023

Add DPGAnalysis/HcalNanoAOD

4ae3163

DryRun force-pushed the CMSSW_12_6_0_pre4_hcalnano branch from 5c086e1 to 4ae3163 Compare January 19, 2023 22:40

cmsbuild added code-checks-pending and removed operations-pending code-checks-rejected labels Jan 19, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 19, 2023

cmsbuild added tests-started and removed tests-pending labels Feb 19, 2023

cmsbuild added tests-approved and removed tests-started labels Feb 20, 2023

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Mar 7, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Mar 7, 2023

Run over TestEnablesEcalHcal and local runs

5938216

DryRun force-pushed the CMSSW_12_6_0_pre4_hcalnano branch from 0b8a497 to 5938216 Compare March 7, 2023 23:05

cmsbuild added code-checks-pending and removed code-checks-rejected labels Mar 7, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 7, 2023

smuzaffar modified the milestones: CMSSW_13_1_X, CMSSW_13_2_X May 4, 2023

DryRun closed this May 10, 2023

DryRun deleted the CMSSW_12_6_0_pre4_hcalnano branch May 10, 2023 05:02

DryRun restored the CMSSW_12_6_0_pre4_hcalnano branch May 10, 2023 05:03

DryRun mentioned this pull request May 10, 2023

Add prototype HcalNanoAOD #41607

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prototype HcalNanoAOD #40489

Add prototype HcalNanoAOD #40489

DryRun commented Jan 11, 2023 •

edited

Loading

cmsbuild commented Jan 11, 2023

cmsbuild commented Jan 18, 2023

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

mariadalfonso commented Jan 25, 2023

perrotta commented Jan 25, 2023

perrotta commented Jan 25, 2023

cmsbuild commented Jan 25, 2023

cmsbuild commented Feb 20, 2023

vlimant commented Feb 20, 2023

cmsbuild commented Mar 7, 2023

cmsbuild commented Mar 7, 2023

cmsbuild commented Mar 7, 2023

DryRun commented Mar 7, 2023

vlimant commented Apr 3, 2023

DryRun commented May 10, 2023 •

edited

Loading

Add prototype HcalNanoAOD #40489

Add prototype HcalNanoAOD #40489

Conversation

DryRun commented Jan 11, 2023 • edited Loading

PR description:

PR validation:

Other

cmsbuild commented Jan 11, 2023

cmsbuild commented Jan 18, 2023

cmsbuild commented Jan 19, 2023

cmsbuild commented Jan 19, 2023

mariadalfonso commented Jan 25, 2023

perrotta commented Jan 25, 2023

perrotta commented Jan 25, 2023

cmsbuild commented Jan 25, 2023

cmsbuild commented Feb 20, 2023

Comparison Summary

NANO Comparison Summary

vlimant commented Feb 20, 2023

cmsbuild commented Mar 7, 2023

cmsbuild commented Mar 7, 2023

cmsbuild commented Mar 7, 2023

DryRun commented Mar 7, 2023

vlimant commented Apr 3, 2023

DryRun commented May 10, 2023 • edited Loading

DryRun commented Jan 11, 2023 •

edited

Loading

DryRun commented May 10, 2023 •

edited

Loading