TPC: Adding streamer for FastTransform in reconstruction

[matthias-kleiner]

SpaceCharge: adding option to use analytical formula for corrections and distortions

• adding simple default analytical distortions/corrections
• add streamer in digitizer

DebugStreamer: Allow writing out streams in parallel

[davidrohr]

What is sizeof(o2::utils::DebugStreamer)? Is it guaranteed to be the same on CPU and GPU? Otherwise it will break GPU reconstruction.

[matthias-kleiner]

What is sizeof(o2::utils::DebugStreamer)? Is it guaranteed to be the same on CPU and GPU? Otherwise it will break GPU reconstruction.

The class is only defined when DEBUG_STREAMER was set during compiling o2. If it was not specified then the class should be always empty and the same size for CPU and GPU.
If o2 was compiled with DEBUG_STREAMER and one runs on GPU, then I guess it would be of different size, but I will check what happens.

[davidrohr]

Ok, thx. Then it should be fine. However, I propose we add a check at the initialization of the GPUReconstruction class when the DEBUG STREAMER define is set, and if so we throw a fatal if the backend is not CPU, to avoid confusing crashes if accidentally they are used together.

[wiechula]

@matthias-kleiner , thanks for the implementation!
Concerning the streamers, could you please also add then here:
https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L657
https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L668
These functions are also called in reconstruction. @davidrohr are there others needed?

[matthias-kleiner]

Ok, thx. Then it should be fine. However, I propose we add a check at the initialization of the GPUReconstruction class when the DEBUG STREAMER define is set, and if so we throw a fatal if the backend is not CPU, to avoid confusing crashes if accidentally they are used together.

This would be good. Can you implement this?

[shahor02]

@matthias-kleiner did you check if the streaming to trees (even different ones) works in multithreaded code? The root I/O is in general not thread safe, see e.g. https://root-forum.cern.ch/t/root-i-o-and-thread-safety/5857

[davidrohr]

Sure, will take care

…

On 9 February 2023 13:22:37 CET, Matthias Kleiner ***@***.***> wrote: > Ok, thx. Then it should be fine. However, I propose we add a check at the initialization of the GPUReconstruction class when the DEBUG STREAMER define is set, and if so we throw a fatal if the backend is not CPU, to avoid confusing crashes if accidentally they are used together. This would be good. Can you implement this?

[matthias-kleiner]

@matthias-kleiner did you check if the streaming to trees (even different ones) works in multithreaded code? The root I/O is in general not thread safe, see e.g. https://root-forum.cern.ch/t/root-i-o-and-thread-safety/5857

Yes I checked this and it is working fine for me. If each thread gets its unique TFile then it is working without problems.

[matthias-kleiner]

Just some information on how to use it, in case someone wants to use it

1. Create the space-charge object which can be used in the digitizer and the fast transform which can be used in the reconstruction (default analytical formulas are used, in case of custom formulas they should be set in the createTPCSpaceChargeCorrectionAnalytical() function)

.L O2/Detectors/TPC/reconstruction/macro/createTPCSpaceChargeCorrection.C+
createTPCSpaceChargeCorrectionAnalytical("tpctransform_ana.root");

Output of debug spline object:

r fastTransformUnitTest_debug1_gridsize180-129-129.root
fastTransform -> Draw("((lx*50-254.5)/49-lx) : lx","","colz");

corrections local X
corrections local Y
corrections local Z
formula - corrections local X

2. Running the digitizer with the analytical distortions and create debug output

o2-sim-digitizer-workflow --onlyDet TPC --distortionType 2 --readSpaceCharge "distortions_analytical.root" --configKeyValues "DebugStreamerParam.StreamLevel=32;" --tpc-lanes 4 -b

3. Running the reconstruction

mkdir -p TPC/Calib/CorrectionMap
cp tpctransform_ana.root TPC/Calib/CorrectionMap/snapshot.root
o2-tpc-reco-workflow --output-type "tracks" --infile "tpcdigits.root" --condition-remap "file:///data/mkleiner/tmp/anasc/sim/=TPC/Calib/CorrectionMap" --configKeyValues "DebugStreamerParam.StreamLevel=8;" -b

4. In case debug output was created merge the TTrees

hadd debug_fasttransform.root  debug_fasttransform*
root.exe
o2::utils::DebugStreamer::mergeTrees("debug_fasttransform.root", "debug_fasttransform_merged.root");

5. Comparing output corrections with expectation:

root.exe debug_fasttransform_merged.root

corrections local X - formula:tree_0 -> Draw("dx - ((lx*50-254.5)/49-lx) : lx","","");
corrections local Y - formula: tree_0 -> Draw("(lyT -ly) - (-(2 + 0.01 * (lx + (lx * 50 - 254.5) / 49 - lx))) : lx","","*")
corrections local Z - formula: tree_0 -> Draw("(lzT - lz) - (lz * 50 / 51 - lz) : lz","","*")

[matthias-kleiner]

@matthias-kleiner , thanks for the implementation! Concerning the streamers, could you please also add then here: https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L657 https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L668 These functions are also called in reconstruction. @davidrohr are there others needed?

Thanks, I just added them

[wiechula]

@matthias-kleiner , thanks for the implementation! Concerning the streamers, could you please also add then here: https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L657 https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L668 These functions are also called in reconstruction. @davidrohr are there others needed?

Thanks, I just added them

My idea was to add streamers directly in the functions above, as in the Transform to see the values which are called.

[matthias-kleiner]

@matthias-kleiner , thanks for the implementation! Concerning the streamers, could you please also add then here: https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L657 https://github.com/AliceO2Group/AliceO2/blob/dev/GPU/TPCFastTransformation/TPCFastTransform.h#L668 These functions are also called in reconstruction. @davidrohr are there others needed?

Thanks, I just added them

My idea was to add streamers directly in the functions above, as in the Transform to see the values which are called.

Now I understand, I added this

[alibuild]

Error while checking build/O2/fullCI for 0d625ba at 2023-02-10 12:16:

## sw/BUILD/O2-latest/log
/sw/SOURCES/O2/10739-slc8_x86-64/0/GPU/TPCFastTransformation/TPCFastTransform.h(120): error: calling a __device__ function("o2::utils::DebugStreamer::flush() const") from a __host__ function("flushStreamer") is not allowed
ninja: build stopped: subcommand failed.

Full log here.