Multiple files as input of EventSource

[aleberti]

Currently, giving more than one file as input_url to EventSource (specifically I am using the MAGICEventSource from https://github.com/cta-observatory/ctapipe_io_magic) e.g. passing a string with wildcards, throws an error (a TraitError to be precise).

Is there a way, if the logic allows, to pass more files at a time? The issue I have is that in MAGIC data files (at calibrated level; one file per subrun, and each run has multiple subruns) the pointing information from the drive reports are stored in the same file as the events, and sometimes happens that a given subrun has only one drive report available (this can happen for different reasons e.g. the subrun is too short, usually they are ~1min long and have ~7-8 reports available). This prevents to interpolate the pointing position of the events starting from the drive reports.

Therefore, if I could be able to pass all the subruns belonging to the same run to the EventSource, this problem would be no longer there. Also, this would improve the interpolation, since performing the interpolation subrun by subrun will result in extrapolation at the edges of each subrun.

Have you any suggestion on how to do this without introducing changes to the EventSource class? Or do you suggest to adopt a solution similar to ctapipe_io_lst i.e. having the possibility to give as input a file containing the pointing information from the drive reports? In the latter case it would mean for us to have a pre-processing of the files before the stage1 process (MAGIC calibrated-->DL1) to get the drive reports and create this file to give as input to the MAGICEventSource.

Thanks!

[maxnoe]

@aleberti Actually, for ctapipe_io_lst we do need to read multiple files as well, the four parallel streams of LST.

Given as input_url LST-1.1.Runxxxx.yyyyy.fits.fz, the source looks for LST-1.{2,3,4}.Runxxxx.yyyyy.fits.fz at the same path to read all files (here in parallel).

You could do the same for the magic event source, give the path to the first subrun as input_url and let the event source find the other subruns automatically.

[aleberti]

@aleberti Actually, for ctapipe_io_lst we do need to read multiple files as well, the four parallel streams of LST.

Given as input_url LST-1.1.Runxxxx.yyyyy.fits.fz, the source looks for LST-1.{2,3,4}.Runxxxx.yyyyy.fits.fz at the same path to read all files (here in parallel).

You could do the same for the magic event source, give the path to the first subrun as input_url and let the event source find the other subruns automatically.

Ok, I did not notice that! Thank, I will try that then.

[maxnoe]

But I think the general issue is valid. The base class already makes the assumption there always has to be one input_url and that input_url refers to a single existing file (or file that can be downloaded from a url or that is in the dataset path).

That is very much ingrained in the EventSource plugin system, as that looks for an EventSource subclass that is_compatible(input_url).

Changing that is not possible I think, the design of everything is quite build around the fact that you give an input_url from which the event source to be used can be determined.

Maybe we should relax the requirements the EventSource base class is already checking:

input_url = Path(
    directory_ok=False,
    exists=True,
    help="Path to the input file containing events.",
).tag(config=True)

Requiring an existing file in the base class seems very restrictive towards what event sources can do.