Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating replay for 12.2.1 release on runs 345755,346062,346512,347028 #4644

Closed
wants to merge 5 commits into from

Conversation

rappoccio
Copy link
Contributor

Replay Request

Requestor
Sal Rappoccio for PPD

Describe the configuration

  • Release: 12_2_1
  • Run: 345755,346062,346512,347028
  • GTs:
    • expressGlobalTag: 122X_dataRun3_Express_Candidate_2022_02_11_12_52_11
    • promptrecoGlobalTag: 122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14
    • alcap0GlobalTag: 122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14
  • Additional changes:

Purpose of the test
Replay in preparation of MWGR datataking next week.

T0 Operations HyperNews thread
If necessary, provide a link to the HN thread announcing the test to the relevant groups.
Tier0 Operations HN Forum

@francescobrivio
Copy link
Contributor

run replay please

@francescobrivio
Copy link
Contributor

@germanfgv @jhonatanamado the failure in the tests comes from
[DMWM-T0-pylint-test #419](https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-T0-pylint-test/419/) completed. Result was FAILURE
which leads to
OSError: [Errno 2] No such file or directory from https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-T0-pylint-test/419/console
Can you help us out in understanding it? Thanks a lot!

@germanfgv
Copy link
Contributor

Hi @francescobrivio. Right now we are building a new T0 release. The one that will be used during the MWGR
it should be ready in around an hour and then we will trigger the replay using it. Is that ok?

@francescobrivio
Copy link
Contributor

Hi @francescobrivio. Right now we are building a new T0 release. The one that will be used during the MWGR it should be ready in around an hour and then we will trigger the replay using it. Is that ok?

perfect! thanks a lot @germanfgv!

@germanfgv
Copy link
Contributor

run replay please

@germanfgv
Copy link
Contributor

It sseems there is a problem with the container using the new version of T0. I'm starting the replay manually while we debug the issue

@germanfgv
Copy link
Contributor

It sseems there is a problem with the container using the new version of T0. I'm starting the replay manually while we debug the issue

I started the manual replay. I'll let you know if we find any issues.

@germanfgv
Copy link
Contributor

germanfgv commented Feb 12, 2022

So far we only have an issue with ExpressAligment. There is a "NoRecord" error regarding PCastorRcd as described here:

Fatal Exception (Exit Code: 8001)
An exception of category 'NoRecord' occurred while
   [0] Processing  Event run: 346512 lumi: 104 event: 101103647 stream: 4
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module TrackingRecoMaterialAnalyser/'materialDumperAnalyzer'
   [3] Prefetching for module DuplicateListMerger/'generalTracks'
   [4] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [5] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [6] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [7] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [8] Prefetching for module TrackProducer/'initialStepTracks'
   [9] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [10] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [11] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [12] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [13] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [14] Prefetching for module CandPtrSelector/'jetsForCoreTrackingPreSplitting'
   [15] Prefetching for module FastjetJetProducer/'ak4CaloJetsForTrkPreSplitting'
   [16] Prefetching for EventSetup module CaloGeometryBuilder/''
   [17] Calling method for EventSetup module CastorGeometryFromDBEP/''
   [18] While getting dependent Record from Record CastorGeometryRecord
Exception Message:
No "PCastorRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.

Is this a GT related error? @francescobrivio @rappoccio

Tarball can be found in

 /afs/cern.ch/user/c/cmst0/public/PausedJobs/MWGR1-2022/Job_4015/tarball

@francescobrivio
Copy link
Contributor

Is this a GT related error? @francescobrivio @rappoccio

Hi @germanfgv apparently there is still some code in the new 12_2_1 release that still consumes the Castor tag (although Castor has been decommissioned for Run3). We'll fix the GT to add back the Castor tag and come back at you asap.

@francescobrivio
Copy link
Contributor

Hi @germanfgv, here are the two new candidate GTs:

Express: 122X_dataRun3_Express_Candidate_2022_02_12_13_35_47
Prompt:  122X_dataRun3_Prompt_Candidate_2022_02_12_13_37_24

Could you re-trigger the replay with these?

@boudoul
Copy link

boudoul commented Feb 12, 2022

Hello Please announce replays (and report issues..) on Tier0 - HN -- Thank you.

@mmusich
Copy link
Contributor

mmusich commented Feb 12, 2022

In this configuration I see:

alcaTrackingOnlyScenario = "trackingOnlyEra_Run2_2018"

This presumably brings in the castor dependence in the CaloGeometryProducer. Would suggest to move to a Run3-based configuration instead

@germanfgv
Copy link
Contributor

@mmusich Yes marco, I'm trying to run right now with using trackingOnlyEra_Run3. Thank you for pointing that out.

@tvami
Copy link
Contributor

tvami commented Feb 12, 2022

Hi @germanfgv did you try it outside of this PR? I just didnt see the commit that changed back the GT and the change of Era

@germanfgv
Copy link
Contributor

Yes @tvami, I manually launched a replay making the change Marco suggested. In it, ExpressAligment workflow for 346512 is almost done, so it seems the fix worked. I will update the configuration in this PR.

@tvami
Copy link
Contributor

tvami commented Feb 12, 2022

Hi @germanfgv since everything seems fine, I converted the candidate GTs into versioned GTs:

122X_dataRun3_HLT_v3
122X_dataRun3_Express_v3
122X_dataRun3_Prompt_v3

@germanfgv
Copy link
Contributor

The replay has finished successfully. I took a bit longer that expected due to an issue with our agent that prevented some jobs to be submitted until today. I have updated the ReplayConfig file to the one that was used in the replay

expressGlobalTag = "120X_dataRun3_Express_v2"
promptrecoGlobalTag = "120X_dataRun3_Prompt_v2"
alcap0GlobalTag = "120X_dataRun3_Prompt_v2"
expressGlobalTag = "122X_dataRun3_Express_Candidate_2022_02_11_12_52_11"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
expressGlobalTag = "122X_dataRun3_Express_Candidate_2022_02_11_12_52_11"
expressGlobalTag = "122X_dataRun3_Express_v3"

promptrecoGlobalTag = "120X_dataRun3_Prompt_v2"
alcap0GlobalTag = "120X_dataRun3_Prompt_v2"
expressGlobalTag = "122X_dataRun3_Express_Candidate_2022_02_11_12_52_11"
promptrecoGlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
promptrecoGlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
promptrecoGlobalTag = "122X_dataRun3_Prompt_v3"

alcap0GlobalTag = "120X_dataRun3_Prompt_v2"
expressGlobalTag = "122X_dataRun3_Express_Candidate_2022_02_11_12_52_11"
promptrecoGlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
alcap0GlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
alcap0GlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
alcap0GlobalTag = "122X_dataRun3_Prompt_v3"

@tvami
Copy link
Contributor

tvami commented Feb 15, 2022

Hi @rappoccio since this finished successfully, I suggest you close this PR. A follow-up has been merged already: #4645

@rappoccio rappoccio closed this Mar 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants