IDCs: Add option to write IDCDelta to calibration file

[matthias-kleiner]

No description provided.

[shahor02]

Hi @matthias-kleiner
Some fixes are needed, see below.

[shahor02]

The imageIDCDelta is already a TFile (memory based), one cannot write it to the TFile, instead one should write it as a binary data:

if (mDumpIDCDeltaCalibData && mCalibFileDir != "/dev/null") {
  std::string calibFName = fmt::format("{}IDCDelta_side{}_cal_data_{}.root", mCalibFileDir, sideStr,  ccdbInfoIDCDelta.getStartValidityTimestamp());
try {
 std::ofstream calFile (calibFName, ios::out | ios::binary);
 calFile.write(imageIDCDelta->data(), imageIDCDelta->size());
 calFile.close();
} catch (std::exception const& e) {
  LOG(error) << "Failed to store IDC calibration data file " << calibFName << ", reason: " << e.what();
}

[shahor02]

MetaData is a text file, should be created as

if (mMetaFileDir != "/dev/null") {
  o2::dataformats::FileMetaData calMetaData;
  calMetaData.fillFileData(calibFName);
  calMetaData.setDataTakingContext(mDataTakingContext);
  calMetaData.type = "calib;
  calMetaData.priority = "low";
  auto metaFileNameTmp = fmt::format("{}{}.tmp", mMetaFileDir, calibFName);
  auto metaFileName = fmt::format("{}{}.done", mMetaFileDir, calibFName);
  try {
      std::ofstream metaFileOut(metaFileNameTmp);
      metaFileOut << ctfMetaData;
      metaFileOut.close();
      std::filesystem::rename(metaFileNameTmp, metaFileName);
  } catch (std::exception const& e) {
     LOG(error) << "Failed to store CTF meta data file " << metaFileName << ", reason: " << e.what();
  }
}

One would need to add the following:

1. fetching mDataTakingContext: add at

   AliceO2/Detectors/TPC/workflow/include/TPCWorkflow/TPCFactorizeIDCSpec.h

   Line 191 in 3461007

   bool mEnableWritingPadStatusMap{false}; ///< do not store the pad status map in the CCDB

o2::framework::DataTakingContext mDataTakingContext{};

and at

AliceO2/Detectors/TPC/workflow/include/TPCWorkflow/TPCFactorizeIDCSpec.h

Line 89 in 3461007

mTFFirst = currTF;

:

mDataTakingContext = pc.services().get<DataTakingContext>();

2. passing output directories for calib data and metafile:
   add at

   AliceO2/Detectors/TPC/workflow/include/TPCWorkflow/TPCFactorizeIDCSpec.h

   Line 169 in 3461007

   unsigned int mProcessedCRUs{}; ///< number of processed CRUs to keep track of when the writing to CCDB etc. will be done

   data members

std::string mMetaFileDir{};
std::string mCalibFileDir{};

add at

AliceO2/Detectors/TPC/workflow/include/TPCWorkflow/TPCFactorizeIDCSpec.h

Line 429 in 3461007

{"add-offset-for-CCDB-timestamp", VariantType::Bool, false, {"Add an offset of 1 hour for the validity range of the CCDB objects"}},

            {"output-dir", VariantType::String, "none", {"calibration files output directory, must exist"}},
            {"meta-output-dir", VariantType::String, "/dev/null", {"calibration metadata output directory, must exist (if not /dev/null)"}},

and at

AliceO2/Detectors/TPC/workflow/include/TPCWorkflow/TPCFactorizeIDCSpec.h

Line 65 in 3461007

mDisableIDCDelta = ic.options().get<bool>("disable-IDCDelta");

mCalibFileDir = ic.options().get<std::string>("output-dir");
if (mCalibFileDir != "/dev/null") {
  mCalibFileDir = o2::utils::Str::rectifyDirectory(mCalibFileDir);
}
mMetaFileDir = ic.options().get<std::string>("meta-output-dir");
if (mMetaFileDir != "/dev/null") {
  mMetaFileDir = o2::utils::Str::rectifyDirectory(mMetaFileDir);
}

[matthias-kleiner]

Hi @matthias-kleiner Some fixes are needed, see below.

Hi @shahor02
many thanks for your detailed help. I implemented the changes and tested it locally. It creates several files now like:

IDCDelta_sideA_cal_data_1668222565120.root
.ls
KEY: o2::tpc::IDCDelta<unsigned char>	ccdb_object;1	object title

IDCDelta_sideA_cal_data_1668222565120.root.done
LHCPeriod: NOV
run: 0
lurl: /data/mkleiner/tmp/IDCDelta_sideA_cal_data_1668222565120.root
type: calib
ctime: 1668270541446
size: 8385536
md5: 74f623f10e86acf026b72e6da1e3cfb4
priority: low
det_composition: unknown

So I assume it is working.

[shahor02]

Thanks, looks fine.
In the real workflow on the EPN these files will go to dedicated directories.
@ole, where do you set the directories for residual files? I don't see --output-dir and --meta-output-dir set there.