Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_hdf5_datalevels seems broken #2450

Closed
Tobychev opened this issue Nov 6, 2023 · 4 comments · Fixed by #2451
Closed

get_hdf5_datalevels seems broken #2450

Tobychev opened this issue Nov 6, 2023 · 4 comments · Fixed by #2451
Labels

Comments

@Tobychev
Copy link
Contributor

Tobychev commented Nov 6, 2023

Describe the bug
The function get_hdf5_datalevels reports no data levels for files that clearly contains data level data

To Reproduce

import ctapipe.io as cio
import pathlib
g_meeting_file_name = pathlib.Path("../2023 Oct F2F/2023-10-datapipe-workshop-material/data/model_training/gamma_20deg_0deg___cta-prod5b-lapalma_desert-2158m-LaPalma-dark_cone10.alpha_train_cl_merged.DL2.h5")
cio.get_hdf5_datalevels(g_meeting_file_name)

returns ()

Expected behavior
That it returns at least DataLevel.DL1_PARAMETERS

@maxnoe
Copy link
Member

maxnoe commented Nov 6, 2023

Checking with ctapipe-fileinfo, the same file claims to contain images, which is also wong. Seems the merge file didn't adapt the metadata correctly to the options it was given:

        PRODUCT:
            CREATION:
                TIME: '2023-04-18 06:52:01.624'
            DATA:
                ASSOCIATION: Subarray
                CATEGORY: Sim
                LEVELS: DL1_IMAGES,DL1_PARAMETERS,DL2

@maxnoe
Copy link
Member

maxnoe commented Nov 6, 2023

Ok, so get_hdf5_datalevels actually inspects the groups present in the hdf5 file, not the metadata.

@maxnoe
Copy link
Member

maxnoe commented Nov 6, 2023

Aha! The error here is a missing type check.

get_hdf5_metadata expects a tables.File as argument. In your example, you gave a pathlib.Path, which also happens to have a .root attribute, but which is the empty string, so all checks foo in h5file.root return False:

In [4]: f = tables.open_file("../data/datapipe_workshop/model_training/gamma_20deg_0deg___cta-prod5b-lapalma_desert-2158m-LaPalma-dark_cone10.
   ...: alpha_train_cl_merged.DL2.h5")

In [5]: get_hdf5_datalevels(f)
Out[5]: (<DataLevel.DL1_PARAMETERS: 6>, <DataLevel.DL2: 7>)

@Tobychev
Copy link
Contributor Author

Tobychev commented Nov 6, 2023

Aha! I think a secondary issue is how things are presented in the docs, where get_hdf5_metadata comes just after the generic read and write functions that do take just a filename.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants