Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAQ: write throttling in output modules (12_3_X) #36842

Merged
merged 1 commit into from
Feb 7, 2022
Merged

DAQ: write throttling in output modules (12_3_X) #36842

merged 1 commit into from
Feb 7, 2022

Conversation

smorovic
Copy link
Contributor

PR description:

  • More aggresive throtting in case a marker is present in local ramdisk.
    Provides more fine-grained throttling avoiding chance to reach "no disk space" errors induced by latency in applying the throttling.
  • updated unit tests to use new global evf output module and remove test for the deprecated one.

PR validation:

Patch was tested in the global HLT syste (EvFOutputModule).
Also part of the unit test which runs automatically as part of cmsbuild tests. Running it manually, the test passes.

@cmsbuild cmsbuild added this to the CMSSW_12_3_X milestone Jan 31, 2022
@cmsbuild
Copy link
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36842/28035

  • This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

@smorovic
- check throttling also in the output module on each event.
8694d32 
This reduces the chance to overshoot and fill the disk even if check is
done periodically in the input source.
- updated unit tests to use new global evf output module
@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36842/28037

  • This PR adds an extra 16KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @smorovic (Srecko Morovic) for master.

It involves the following packages:

  • EventFilter/Utilities (daq, reconstruction)

@emeschi, @clacaputo, @cmsbuild, @slava77, @jpata, @smorovic can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@smorovic
Copy link
Contributor Author

@cmsbuild please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de2f8c/22101/summary.html
COMMIT: 8694d32
CMSSW: CMSSW_12_3_X_2022-01-30-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36842/22101/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de2f8c/22101/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de2f8c/22101/git-merge-result

RelVals-INPUT

  • 134.911134.911_RunSinglePh2015D+RunSinglePh2015D+HLTDR2_25ns+RECODR2_25nsreHLT_HIPM+HARVESTDR2/step2_RunSinglePh2015D+RunSinglePh2015D+HLTDR2_25ns+RECODR2_25nsreHLT_HIPM+HARVESTDR2.log

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 888 differences found in the comparisons
  • DQMHistoTests: Total files compared: 43
  • DQMHistoTests: Total histograms compared: 3449612
  • DQMHistoTests: Total failures: 6782
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3442807
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.004 KiB( 42 files compared)
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 181 log files, 42 edm output root files, 43 DQM output files
  • TriggerResults: no differences found

@smorovic
Copy link
Contributor Author

smorovic commented Feb 4, 2022

@cmsbuild please test

@jpata
Copy link
Contributor

jpata commented Feb 4, 2022

unassign reconstruction

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2022

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-de2f8c/22227/summary.html
COMMIT: 8694d32
CMSSW: CMSSW_12_3_X_2022-02-04-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36842/22227/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 46
  • DQMHistoTests: Total histograms compared: 3766018
  • DQMHistoTests: Total failures: 2
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3765994
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 193 log files, 42 edm output root files, 46 DQM output files
  • TriggerResults: no differences found

@smorovic
Copy link
Contributor Author

smorovic commented Feb 4, 2022

+1

@jpata
Copy link
Contributor

jpata commented Feb 7, 2022

+reconstruction

  • EventFilter/Utilities is changed in non-reco parts, so reco didn't do any validation here

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 7, 2022

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

perrotta commented Feb 7, 2022

+1

@cmsbuild cmsbuild merged commit 9e42ef7 into cms-sw:master Feb 7, 2022
This was referenced Feb 7, 2022
Merged
Merged
Merged
@smorovic smorovic deleted the 123X-throttle-output-module branch August 24, 2023 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
code-checks-approved daq-approved fully-signed orp-approved reconstruction-approved tests-approved
Projects
None yet
Milestone
CMSSW_12_3_X
Development

Successfully merging this pull request may close these issues.
4 participants
@smorovic @cmsbuild @jpata @perrotta