New DAQ source for L1Trigger scouting #43467

Mmiglio · 2023-12-01T09:55:43Z

PR description:

This PR introduces a new DAQ source for the L1Trigger Scouting Run3 demonstrator.
The L1 Data Scouting (DS) system operates in parallel with respect to the central DAQ. Therefore, L1 DS data are not inserted in the CMS event data, ensuring complete Independence of the two systems.
By design, L1 Scouting data are collected and processed based on the orbit counter. Hence, one CMSSW event correspond to data collected during one LHC orbit (3564 BX).
An overview of this system can be found in this presentation.

The main components introduced in this PR include:

A DAQSourceModel that reads data from all scouting sources and merges them based on the Orbit counter.
Unpackers for the two sources currently tested during Run3 (uGMT and Calo DeMux).
Data formats for the scouting objects.

PR validation:

The system has been tested from two perspectives:

Verified that events collected in the Zero Bias dataset match the raw data collected and unpacked from the L1DS.
Tested online data tacking using DAQ3VAL resources (scout-rubu -> d3v-fu (filter units) -> d3v lustre + monitoring).

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

No backport needed.

Implemented a new DAQSourceModel to read and merge Run3 L1 scouting data. - Similar to the existing FRDStriped input source - Modified EvFDaqDirector to enable the possibility of having multiple data sources per ramdisk - Added DataFormats/L1Scouting containing Scouting Raw Data Collection - Vector of FEDRawData - Included optional argument wordSize in the FEDRawData resize methods: 8 bytes by default, can be set to 4 bytes for scouting - Not a real FED, using a SRDCollection since L1scouting data will never be mixed with event data - Removed old scouting files that were used as an example - Test script for the ScoutingRun3 daq source

…mpression factor of 2

…d to it. Fixed bug in the bxOffset index

Implemented a new DAQSourceModel to read and merge Run3 L1T scouting raw data. - Similar to the existing FRDStriped input source - Modified EvFDaqDirector to enable the possibility of having multiple data sources per ramdisk - Added DataFormats/L1Scouting containing Scouting Raw Data Collection - Vector of FEDRawData - Included optional argument wordSize in the FEDRawData resize methods: 8 bytes by default, can be set to 4 bytes for scouting - Not a real FED, using a SRDCollection since L1scouting data will never be mixed with event data - Removed old scouting files that were used as an example Added first two unpackers for uGMT and CaloL2 data - Custom data format, similar to l1t objects, but more efficent in term of storage and write speed. - Save 30% in storage - Objects stored in an OrbitCollection Example of dummy analyzer used to dump scouting data - consumes collections produced by unpackers and dump them to screen

cmsbuild · 2023-12-01T10:01:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43467/38020

This PR adds an extra 292KB to repository
Found files with invalid states:
- EventFilter/Utilities/test/testScoutingRun3_daqsource.py:
  - Added: 8067ab5
  - Deleted: 2f1a7c5
- DataFormats/L1Scouting/interface/SDSRawDataCollection.h:
  - Added: 8067ab5
  - Deleted: 12b08e1
- L1TriggerScouting/Utilities/plugins/tmp.out:
  - Added: 4c0e625
  - Deleted: b55f10b
- DataFormats/L1Scouting/interface/SDSNumbering.h:
  - Added: 8067ab5
  - Deleted: 12b08e1
- EventFilter/L1ScoutingRawToDigi/interface/scales.h:
  - Added: 9029aab
  - Modified: 1771948, 4949fc8
  - Deleted: 2f1a7c5
- DataFormats/L1Scouting/src/SDSRawDataCollection.cc:
  - Added: 8067ab5
  - Deleted: 12b08e1
- EventFilter/L1ScoutingRawToDigi/interface/conversion.h:
  - Added: 1771948
  - Modified: 4949fc8, afe8b59
  - Deleted: 2f1a7c5
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/FEDRawData/interface/FEDRawData.h modified in PR(s): [DAQ] Define missing assignment operator to fix warnings reported in DEVEL IBs #43461

cmsbuild · 2023-12-01T10:02:17Z

A new Pull Request was created by @Mmiglio (Matteo Migliorini) for master.

It involves the following packages:

DataFormats/FEDRawData (daq)
DataFormats/L1Scouting (****)
DataFormats/L1ScoutingRawData (****)
EventFilter/L1ScoutingRawToDigi (****)
EventFilter/Utilities (daq)
L1TriggerScouting/Utilities (****)

The following packages do not have a category, yet:

DataFormats/L1Scouting
DataFormats/L1ScoutingRawData
EventFilter/L1ScoutingRawToDigi
L1TriggerScouting/Utilities
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@smorovic, @emeschi, @cmsbuild can you please review it and eventually sign? Thanks.
@missirol, @rovere, @Martin-Grunewald this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

smorovic · 2023-12-01T10:23:22Z

@cmsbuild please test with #43461

cmsbuild · 2023-12-01T10:49:48Z

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5c4eca/36241/summary.html
COMMIT: 9dd2a02
CMSSW: CMSSW_14_0_X_2023-11-30-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43467/36241/install.sh to create a dev area with all the needed externals and cmssw changes.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

cmsbuild · 2023-12-15T17:40:18Z

Pull request #43467 was updated. @emeschi, @smorovic, @cmsbuild can you please check and sign again.

smorovic · 2023-12-16T08:35:56Z

please test

cmsbuild · 2023-12-16T11:37:12Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5c4eca/36527/summary.html
COMMIT: 9d13a12
CMSSW: CMSSW_14_0_X_2023-12-15-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43467/36527/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 67 lines to the logs
Reco comparison results: 20 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3429858
DQMHistoTests: Total failures: 1202
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3428634
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

smorovic · 2023-12-17T19:00:13Z

+daq

cmsbuild · 2023-12-17T19:00:47Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2023-12-19T15:37:51Z

+1

makortel

Given that the PR was already merged, I could also open an issue to follow up on these points (it was just easy to comment here). They should be addressed before I'd be comfortable for the RAW-level backward compatibility guarantee.

makortel · 2023-12-19T15:57:50Z

DataFormats/L1Scouting/test/TestL1ScoutingFormat.sh

This test would still need a component where a ROOT file produced with a specific version of CMSSW (and stored in cms-data) is read with the current CMSSW.

What procedure should I follow to include the DataFormats-L1Scouting repository in cms-data? If that's unnecessary, where should I upload the test file?

The procedure is to ask @smuzaffar (or @cms-sw/externals-l2 ?) to create the DataFormats-L1Scouting and DataFormats-L1ScoutingRawData repositories to cms-data, and then make PRs to those repositories to add the test files. The in the CMSSW PR tests the cms-data PRs can be included so that all of them can be tested together.

Thanks!

New cms-data repos can be requested by created issue like #43621 . Once L2 sign the issue then bot will create the cms-data repo

by the way, I have created the issues for these two data repos. We need @cms-sw/daq-l2 to sign them in order to create these repos

Thanks for creating the two repos, I've opened two PRs to add the test files. Once merged, I will add the reading procedure to the unit tests.

makortel · 2023-12-19T15:58:16Z

DataFormats/L1ScoutingRawData/test/TestSDSRawDataCollectionFormat.sh

This test would still need a component where a ROOT file produced with a specific version of CMSSW (and stored in cms-data) is read with the current CMSSW.

makortel · 2023-12-19T17:05:03Z

DataFormats/L1Scouting/interface/L1ScoutingCalo.h

+    int hwIso_;
+  };
+
+  class Jet : public CaloObject {


If separate types are needed, please avoid the inheritance (to keep the structure simpler, and therefore easier to support for future evolution).

If separate types are not needed, how about just using CaloObject directly for the three cases?

Given that persistent data has already been created using these classes as they were in this PR (https://its.cern.ch/jira/browse/CMSTRANSF-799), perhaps it is easiest to just leave the inheritance as it is. The consequence is that the class hierarchy may not be changed in the future. In the PR adding the tests, please add comments here documenting this restriction.

smuzaffar · 2023-12-20T18:32:27Z

We should have squashed/cleaned up this PR , it has added relatively large tmp.out file in the history now :-(

smorovic · 2023-12-20T23:31:03Z

We should have squashed/cleaned up this PR , it has added relatively large tmp.out file in the history now :-(

Was the issue many actual commits, or that there was also one merge within them?
FWIW, I couldn't cleanly rebase that branch to CMSSW_13_X without a lot of conflicts (for running analysis on some data produced with same as this PR), I gave up and just applied a diff in the end.

smuzaffar · 2023-12-21T09:24:24Z

too many commits or merge commit is not an issue. As bot reported in #43467 (comment), there were files which were added and removed which we should avoid (tmp.out was one so such file)

Mmiglio added 13 commits November 10, 2023 16:24

Renamed L1 scouting raw data data formats

567e1fc

[WIP] fixed orbit collection, new data structure allows to achieve co…

b9c7ba3

…mpression factor of 2

[WIP] added new calo data structures

12b08e1

[WIP] added plugins

9029aab

[WIP] cleaned orbit collection, now an orbit buffer is directly passe…

1771948

…d to it. Fixed bug in the bxOffset index

Added Example analyzer

4c0e625

remove dummy file

b55f10b

fixed small buggs in the unpackers + renamed scouting namespace

4949fc8

New structure for EtSums

afe8b59

Added AaÃòll sums and moved conversion scripts to L1TScouting

2f1a7c5

Applied code-checks and code-format

9dd2a02

cmsbuild added this to the CMSSW_14_0_X milestone Dec 1, 2023

cmsbuild added daq-pending pending-signatures tests-pending orp-pending new-package-pending code-checks-pending labels Dec 1, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 1, 2023

cmsbuild added tests-started requires-external and removed tests-pending labels Dec 1, 2023

cmsbuild removed the tests-started label Dec 1, 2023

cmsbuild added the code-checks-approved label Dec 15, 2023

cmsbuild added tests-started and removed tests-pending labels Dec 16, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 16, 2023

cmsbuild added daq-approved fully-signed and removed daq-pending pending-signatures labels Dec 17, 2023

cmsbuild added orp-approved and removed orp-pending labels Dec 19, 2023

cmsbuild merged commit 16cf9a6 into cms-sw:master Dec 19, 2023
11 checks passed

makortel reviewed Dec 19, 2023

View reviewed changes

This was referenced Dec 19, 2023

[14.0.X] Updated root to tip of branch v6-30-00-patches cms-sw/cmsdist#8896

Closed

Materialize terror symbols cms-sw/root#191

Closed

First PyTorch tests for TorchScript inference CPU/CUDA #43475

Merged

This was referenced Dec 20, 2023

Create repository DataFormats-L1ScoutingRawData #43621

Closed

Create repository DataFormats-L1Scouting #43620

Closed

This was referenced Dec 20, 2023

Add test file for DataFormats/L1ScoutingRawData cms-data/DataFormats-L1ScoutingRawData#1

Merged

Add test file for DataFormats/L1Scouting cms-data/DataFormats-L1Scouting#1

Merged

Mmiglio mentioned this pull request Feb 15, 2024

Update L1Scouting Data Format tests and minor code fixes #43984

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New DAQ source for L1Trigger scouting #43467

New DAQ source for L1Trigger scouting #43467

Mmiglio commented Dec 1, 2023

cmsbuild commented Dec 1, 2023

cmsbuild commented Dec 1, 2023 •

edited

Loading

smorovic commented Dec 1, 2023

cmsbuild commented Dec 1, 2023

cmsbuild commented Dec 15, 2023

smorovic commented Dec 16, 2023

cmsbuild commented Dec 16, 2023

smorovic commented Dec 17, 2023

cmsbuild commented Dec 17, 2023

antoniovilela commented Dec 19, 2023

makortel left a comment

makortel Dec 19, 2023

Mmiglio Dec 20, 2023

makortel Dec 20, 2023

smuzaffar Dec 20, 2023

smuzaffar Dec 20, 2023

Mmiglio Dec 20, 2023

makortel Dec 19, 2023

makortel Dec 19, 2023

makortel Jan 19, 2024

smuzaffar commented Dec 20, 2023

smorovic commented Dec 20, 2023 •

edited

Loading

smuzaffar commented Dec 21, 2023

New DAQ source for L1Trigger scouting #43467

New DAQ source for L1Trigger scouting #43467

Conversation

Mmiglio commented Dec 1, 2023

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Dec 1, 2023

cmsbuild commented Dec 1, 2023 • edited Loading

smorovic commented Dec 1, 2023

cmsbuild commented Dec 1, 2023

Clang Build

cmsbuild commented Dec 15, 2023

smorovic commented Dec 16, 2023

cmsbuild commented Dec 16, 2023

Comparison Summary

smorovic commented Dec 17, 2023

cmsbuild commented Dec 17, 2023

antoniovilela commented Dec 19, 2023

makortel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smuzaffar commented Dec 20, 2023

smorovic commented Dec 20, 2023 • edited Loading

smuzaffar commented Dec 21, 2023

cmsbuild commented Dec 1, 2023 •

edited

Loading

smorovic commented Dec 20, 2023 •

edited

Loading