-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix memory use when writing ParameterSets to files #44727
Conversation
When forwarding a VPSet from the input file to the output, avoid calling vpset() which triggers creating the entire vector.
cms-bot internal usage |
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44727/39937
|
A new Pull Request was created by @Dr15Jones for master. It involves the following packages:
@makortel, @cmsbuild, @Dr15Jones, @smuzaffar can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
@cmsbuild, please test |
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-300972/38822/summary.html Comparison SummarySummary:
|
Looking the So either we are not testing the pathological cases in PR tests, or I didn't happen to look at them. |
+core |
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2) |
+1 |
@Dr15Jones is this fix being backported to old cmssw releases so they work when we retry them in production or do we need to do a dirty fix on our end? |
@jenimal Of which CMSSW release cycles are you particularly interested in? The backports Chris mentioned were down to 13_2_X. For the necessary release cycles we'd also need new releases (@cms-sw/orp-l2), and then somehow those release would have to be used in the merge jobs (but you probably know more about how all that works). |
Hi @makortel, these WFs are using |
Will you be able to use a new 13_2_X release somehow in the existing workflows, or will the fix be applicable only for new workflows? |
So I was checking the CMSSW release that the existing WFs were using. When we do resubmission, it will use the same release. I don't think this is an issue since |
13_2_6_patch2 does not include the backport (well, given that the PR was merged yesterday, none of the 13_2_X releases have the backport). In order to have the fix, we need to build a new 13_2_X release, and somehow the necessary workflows need to use the new release. |
So how a backport works is that, if a fix is backported to a specific set of releases, the fix is only applied to any new release of that set, but not the existing releases? I'm not sure how viable is to change the CMSSW release setting in a job configuration file. I'll need to find out. I'm a bit concerned about the future merge jobs. If we build a new 13_2_X release, will the future WFs use it instead of the old release? Otherwise such issue will still emerge. |
Correct. All already-built releases are immutable.
That is exactly the question I'm wondering too. |
Hi @makortel. When we backported the fix to a specific CMSSW release, do we increment the version number and create a new release version? If so, what is the new release number? Sorry if I have missed it. |
When a new release in a release cycle is built, the version number is indeed increased. But a new release is built only when really needed (i.e. not after every backport), so you didn't miss anything. If I'm not mistaken the next version in 13_2_X would be CMSSW_13_2_11. |
Fair enough. A new release is built when we accumulate enough changes. Is that also when the update is propagated into CVMFS? After that, future workflows can use the new release like CMSSW_13_2_11. |
Correct. I can ask for a new 13_2_X release in the release planning meeting next week (advance warning to @cms-sw/orp-l2). Does the "future workflows can use the new release" need action from PdmV? (let's tag them in any case @cms-sw/pdmv-l2) |
I think the action from our side would be to be sure to use the "cured" releases for any future submission. And eventually ask for a backport of this in case we understand we need it for an older release. |
If backporting it to an older release is allowed, can we do it for 13_2_6_patch2? |
@yanfr0818 I’m afraid releases are immutable once published so the jobs would have to switch to the new release. |
PR description:
When forwarding a VPSet from the input file to the output, avoid calling vpset() which triggers creating the entire vector.
PR validation:
All framework unit tests pass. When this change was applied to a production nanoAOD merge job that was having memory problems (grew to 4GB+) the memory use remained flat (1.9GB).
fixes #44679