Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow tables in arbitrarily deep hdf5 groups #1068

Merged
merged 2 commits into from
May 20, 2019

Conversation

kosack
Copy link
Contributor

@kosack kosack commented May 9, 2019

This is a change to HDF5TableWriter to allow tables to be output to arbitrarily deep groups in an HDF5 file.

The main change (other than some reformatting) is:

  • the group_name attribute of the HDF5TableWriter is now considered the "base" group (from which all sub-tables will be under)
  • writer.write(table_name="x", containers=y) works as before, but now you can add deeper nesting like writer.write(table_name="subgroup/anothergroup/x", containers=y) and subgroup and anothergroup will be created automatically if they don't exist.
  • the unit test test_write_to_any_location has been re-enabled (it was marked as xfail before)

A full example:

with HDF5TableWriter("output.h5", group_name="/dl1/tel") as writer:
    for event in source:
         ...
         writer.write("image/NectarCam", somedata) 
         writer.write("params", paramdata)

Would create 2 tables: /dl1/tel/image/NectarCam and /dl1/tel/params

This allow a nicer structure, as shown in #1059 (v5 and above)

kosack added 2 commits May 9, 2019 16:47
- test_write_to_any_location() no longer is marked as xfail
- "blackened" the code
- group_name is now the base group name of where to write (defaults to "/")
- the table name can contain a hierarchy, e.g. "tel/some_table" and parent groups will be automatically generated (using the `createparents` option of `pytables.create_table()`
@kosack kosack changed the title Feature/improve hdf5tablewriter Allow tables in arbitrary deep hdf5 groups May 9, 2019
@kosack kosack changed the title Allow tables in arbitrary deep hdf5 groups Allow tables in arbitrarily deep hdf5 groups May 9, 2019
@codecov
Copy link

codecov bot commented May 9, 2019

Codecov Report

Merging #1068 into master will increase coverage by 0.05%.
The diff coverage is 98.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1068      +/-   ##
==========================================
+ Coverage   83.44%   83.49%   +0.05%     
==========================================
  Files         181      181              
  Lines       10439    10448       +9     
==========================================
+ Hits         8711     8724      +13     
+ Misses       1728     1724       -4
Impacted Files Coverage Δ
ctapipe/io/tests/test_hdf5.py 97.24% <100%> (+2.38%) ⬆️
ctapipe/io/hdf5tableio.py 93.49% <88.88%> (-0.42%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fea8a57...f7f6789. Read the comment docs.

@vuillaut
Copy link
Member

vuillaut commented May 9, 2019

Definitely something we want.

@kosack kosack requested a review from maxnoe May 10, 2019 16:11
@kosack
Copy link
Contributor Author

kosack commented May 16, 2019

I think this is ready to go - codacy complains a bit about some test variable names, but I think those can be ignored for now.

@kosack kosack merged commit 87bde96 into cta-observatory:master May 20, 2019
@kosack kosack deleted the feature/improve_hdf5tablewriter branch August 21, 2019 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants