Add support for DL3 protons

[HealthyPear]

The tool lstchain_create_dl3_file currently expects at least a simulated source with either a name or coordinates

In case of simulated protons used to create background models (e.g. because of lack of OFF sources for the source we are interested in) we would start from diffuse all-sky simulations of a source which has no name and which "true" coordinates are the ones simulated.

Apologies if this is already possible somehow, I could have overlooked.

[HealthyPear]

I am testing this on a merged DL2 file which I created with the DL1 to DL2 script of lstchain.

If one tries to make the source information not required by default and use the simulated data to compute the sky direction offsets in lstchain.high_lvel.hdu_table.create_event_list then the problem falls down to actually loading the mc dl2 data which fails in because the following quantities change between runs

energy_range_min
energy_range_max
max_alt
min_alt
max_az
min_az
max_viewcone_radius
max_scatter_range

P.S: the DL2 file in question is dl2_20240918_v0.10.12_allsky_nsb_tuning_0.14_train_dec_4822_Protons_merged.h5 so I am using the corresponding version of lstchain.

[moralejo]

A DL3 file should correspond to a GTI (with no significant change of IRF within the run, I think), so using a merged DL2 file with a wide variety of different pointings (hence thresholds, and IRF in general) would be wrong in any case.

I thin that as a start, you should only merge files from the same pointing node (other issues may pop up later).

[HealthyPear]

I did not merge the runs myself, this is file is a DL2 transformation of the DL1 dataset from the IT cluster used as training dataset of another production, which means that models used to produce the IRFs are also trained mixing all this information before hand

if this is not a problem than perhaps lstchain dl2_to_dl3 should deal with this scenario by running on chunks of data with the same properties and then write a set of DL3 files one for each corresponding pointing/IRFs

or there should be a script than "unmerges" the files and runs the DL2-to-DL3 transformation on each chunk

am I correct?

[moralejo]

I did not merge the runs myself, this is file is a DL2 transformation of the DL1 dataset from the IT cluster used as training dataset of another production, which means that models used to produce the IRFs are also trained mixing all this information before hand

That is a different thing. For training you can of course mix, as long as the pointing parameters are among the RF inputs. This allows us to process all data of a given project (~same declination) with the same RF.

To create DL2 and DL3 files from protons you should use test MC, not training MC. We do not have standard protons test MC, so the only solution would be to spare some of the training protons (i.e, not use them in growing the RFs) and reserve them for creating the DL3 with them (on a pointing-by-pointing basis).

What were the RFs used for the dl1 to dl2 transformation? You say "...as training dataset of another production", that is confusing. It is not right to apply the RFs of one production to another (with different parameters).

if this is not a problem than perhaps lstchain dl2_to_dl3 should deal with this scenario by running on chunks of data with the same properties and then write a set of DL3 files one for each corresponding pointing/IRFs

Yes, but the solution is not to do the merging, right?

[HealthyPear]

It is not right to apply the RFs of one production to another (with different parameters).

I know this, but I had not time to produce test protons, so I preferred to go for an approximation and use the protons from a very similar production to make a background model and get a preliminary result.

Yes, but the solution is not to do the merging, right?

yes sorry there should ba a dl3 run for each dl2 run, so run the DL2-to-DL3 transformation on each run pertaining to the same IRF node of the DL2 merged file with the settings used for the IRFs of that node

[HealthyPear]

Still, I think the source name and ra/dec part is relevant: currently the script cannot run on a MC proton file because that is not a source which can be found on catalogs like the script is attempting to do now

[morcuended]

I think you can just give the coordinates, no need for the source name. They are written as metatada in the HDU headers.

[maxnoe]

Still, I think the source name and ra/dec part is relevant: currently the script cannot run on a MC proton file because that is not a source which can be found on catalogs like the script is attempting to do now

Shouldn't the source and ra/dec you give here still be the source you are interested in if the goal is to create a background model for that source?