Creating replay for 12.2.1 release on runs 345755,346062,346512,347028

[rappoccio]

Replay Request

Requestor
Sal Rappoccio for PPD

Describe the configuration

• Release: 12_2_1
• Run: 345755,346062,346512,347028
• GTs:
  • expressGlobalTag: 122X_dataRun3_Express_Candidate_2022_02_11_12_52_11
  • promptrecoGlobalTag: 122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14
  • alcap0GlobalTag: 122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14
• Additional changes:

Purpose of the test
Replay in preparation of MWGR datataking next week.

T0 Operations HyperNews thread
If necessary, provide a link to the HN thread announcing the test to the relevant groups.
Tier0 Operations HN Forum

[francescobrivio]

run replay please

[francescobrivio]

@germanfgv @jhonatanamado the failure in the tests comes from
[DMWM-T0-pylint-test #419](https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-T0-pylint-test/419/) completed. Result was FAILURE
which leads to
OSError: [Errno 2] No such file or directory from https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-T0-pylint-test/419/console
Can you help us out in understanding it? Thanks a lot!

[germanfgv]

Hi @francescobrivio. Right now we are building a new T0 release. The one that will be used during the MWGR
it should be ready in around an hour and then we will trigger the replay using it. Is that ok?

[francescobrivio]

Hi @francescobrivio. Right now we are building a new T0 release. The one that will be used during the MWGR it should be ready in around an hour and then we will trigger the replay using it. Is that ok?

perfect! thanks a lot @germanfgv!

[germanfgv]

run replay please

[germanfgv]

It sseems there is a problem with the container using the new version of T0. I'm starting the replay manually while we debug the issue

[germanfgv]

It sseems there is a problem with the container using the new version of T0. I'm starting the replay manually while we debug the issue

I started the manual replay. I'll let you know if we find any issues.

[germanfgv]

So far we only have an issue with ExpressAligment. There is a "NoRecord" error regarding PCastorRcd as described here:

Fatal Exception (Exit Code: 8001)
An exception of category 'NoRecord' occurred while
   [0] Processing  Event run: 346512 lumi: 104 event: 101103647 stream: 4
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module TrackingRecoMaterialAnalyser/'materialDumperAnalyzer'
   [3] Prefetching for module DuplicateListMerger/'generalTracks'
   [4] Prefetching for module TrackProducer/'mergedDuplicateTracks'
   [5] Prefetching for module DuplicateTrackMerger/'duplicateTrackCandidates'
   [6] Prefetching for module TrackCollectionMerger/'preDuplicateMergingGeneralTracks'
   [7] Prefetching for module TrackCollectionMerger/'earlyGeneralTracks'
   [8] Prefetching for module TrackProducer/'initialStepTracks'
   [9] Prefetching for module MkFitOutputConverter/'initialStepTrackCandidates'
   [10] Prefetching for module MkFitEventOfHitsProducer/'mkFitEventOfHits'
   [11] Prefetching for module MkFitSiPixelHitConverter/'mkFitSiPixelHits'
   [12] Prefetching for module SiPixelRecHitConverter/'siPixelRecHits'
   [13] Prefetching for module JetCoreClusterSplitter/'siPixelClusters'
   [14] Prefetching for module CandPtrSelector/'jetsForCoreTrackingPreSplitting'
   [15] Prefetching for module FastjetJetProducer/'ak4CaloJetsForTrkPreSplitting'
   [16] Prefetching for EventSetup module CaloGeometryBuilder/''
   [17] Calling method for EventSetup module CastorGeometryFromDBEP/''
   [18] While getting dependent Record from Record CastorGeometryRecord
Exception Message:
No "PCastorRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.

Is this a GT related error? @francescobrivio @rappoccio

Tarball can be found in

 /afs/cern.ch/user/c/cmst0/public/PausedJobs/MWGR1-2022/Job_4015/tarball

[francescobrivio]

Is this a GT related error? @francescobrivio @rappoccio

Hi @germanfgv apparently there is still some code in the new 12_2_1 release that still consumes the Castor tag (although Castor has been decommissioned for Run3). We'll fix the GT to add back the Castor tag and come back at you asap.

[francescobrivio]

Hi @germanfgv, here are the two new candidate GTs:

Express: 122X_dataRun3_Express_Candidate_2022_02_12_13_35_47
Prompt:  122X_dataRun3_Prompt_Candidate_2022_02_12_13_37_24

Could you re-trigger the replay with these?

[boudoul]

Hello Please announce replays (and report issues..) on Tier0 - HN -- Thank you.

[mmusich]

In this configuration I see:

T0/etc/ReplayOfflineConfiguration.py

Line 114 in 7b3f99b

alcaTrackingOnlyScenario = "trackingOnlyEra_Run2_2018"

This presumably brings in the castor dependence in the CaloGeometryProducer. Would suggest to move to a Run3-based configuration instead

[germanfgv]

@mmusich Yes marco, I'm trying to run right now with using trackingOnlyEra_Run3. Thank you for pointing that out.

[tvami]

Hi @germanfgv did you try it outside of this PR? I just didnt see the commit that changed back the GT and the change of Era

[germanfgv]

Yes @tvami, I manually launched a replay making the change Marco suggested. In it, ExpressAligment workflow for 346512 is almost done, so it seems the fix worked. I will update the configuration in this PR.

[tvami]

Hi @germanfgv since everything seems fine, I converted the candidate GTs into versioned GTs:

122X_dataRun3_HLT_v3
122X_dataRun3_Express_v3
122X_dataRun3_Prompt_v3

[germanfgv]

The replay has finished successfully. I took a bit longer that expected due to an issue with our agent that prevented some jobs to be submitted until today. I have updated the ReplayConfig file to the one that was used in the replay

[tvami]

Suggested change

	expressGlobalTag = "122X_dataRun3_Express_Candidate_2022_02_11_12_52_11"
	expressGlobalTag = "122X_dataRun3_Express_v3"

[tvami]

Suggested change

	promptrecoGlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
	promptrecoGlobalTag = "122X_dataRun3_Prompt_v3"

[tvami]

Suggested change

	alcap0GlobalTag = "122X_dataRun3_Prompt_Candidate_2022_02_11_12_53_14"
	alcap0GlobalTag = "122X_dataRun3_Prompt_v3"

[tvami]

Hi @rappoccio since this finished successfully, I suggest you close this PR. A follow-up has been merged already: #4645