Change of formats to new DAQ Model version

[morcuended]

While trying to process run1720, I got these errors:

Traceback (most recent call last):
...
  File "/fefs/home/daniel.morcuende/software/lstchain_v2_dev/cta-lstchain/lstchain/reco/dl0_to_dl1.py", line 247, in r0_to_dl1
    for i, event in enumerate(source):
  File "/home/daniel.morcuende/.conda/envs/lstchain_v2_dev/lib/python3.7/site-packages/ctapipe/io/eventsource.py", line 217, in __iter__
    for event in self._generator():
  File "/home/daniel.morcuende/.conda/envs/lstchain_v2_dev/lib/python3.7/site-packages/ctapipe_io_lst/__init__.py", line 153, in _generator
    self.fill_lst_event_container_from_zfile(event)
  File "/home/daniel.morcuende/.conda/envs/lstchain_v2_dev/lib/python3.7/site-packages/ctapipe_io_lst/__init__.py", line 259, in fill_lst_event_container_from_zfile
    unpacked_cdts =  struct.unpack(rec_fmt, event.lstcam.cdts_data)
struct.error: unpack requires a buffer of 31 bytes

This seems to be due to the recent upgrade of DAQ data model (extraction of CDTS data). Now it should be changed according to this structure:

typedef struct
{
    tUInt64 uctsTimeStamp;
    tUInt32 uctsAddress;
    tUInt32 eventCounter;
    tUInt32 busyCounter;
    tUInt32 ppsCounter;
    tUInt32 clockCounter;
    tUInt8 triggerType;                 // For TIB, this is the first 7 bits of SPI
    tUInt8 whiteRabbitResetBusyStatus;  // A few status flags here, see below
    tUInt8 stereoPattern;               // For TIB, this is the next 8 bits of the SPI string
    tUInt8 numInBunch;                  // This information only needed for debugging
    tUInt32 cdtsVersion;                // Should be x.y.z where x is 2bytes and y, z are a byte each
} __attribute__((__packed__)) tExtDevicesCDTSMessage;

[morcuended]

Well...it seems that this only affects to runs from 20191206 because I could process runs from 20191209 without problems.

So false alarm. I will keep this open in case this is useful for anyone else.

[morcuended]

It turns out that only run1720 is affected by this format change.

[morcuended]

I open this issue again.

After discussing it with Isidro, we found out that run1720 was the only one with UCTS timestamps after the first ~1000 events without them. Whenever there is no UCTS data available (i.e. all other runs except for run1720) there is no problem. However, if UCTS data is present, the current data model is not valid, so it must be updated to the new CDTS version: DAQ37827

[FrancaCassol]

Hi @morcuended,
yes, I just discovered it me too, I will add the change as soon as possible in the above

[sizun]

Hi @FrancaCassol, remember that I already made the change for NectarCAM:
cta-observatory/ctapipe_io_nectarcam@8590e05

[rlopezcoto]

Solved in #19