Add condition option to read_table

[maxnoe]

This adds the possibility to use tables read_where method using a numexpr expression to only load rows fulfilling condition.

Very handy for example to load data from only one telescope from a by_type table: condition="tel_id == 1" or only load reconstructed events: condition = "hillas_length > 0"

[kosack]

good idea - pytables' in-kernel filters are very efficient as well, so this should speed up any pre-filtering.

[kosack]

do you know if the start/stop/step apply before or after the filtering? I was going to implement a chunked iterator wrapping read_table, so that may have an effect.

[codecov]

Codecov Report

Merging #1769 (d4319d6) into master (04d3f15) will increase coverage by 0.18%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1769      +/-   ##
==========================================
+ Coverage   92.03%   92.21%   +0.18%     
==========================================
  Files         186      186              
  Lines       14605    15124     +519     
==========================================
+ Hits        13441    13947     +506     
- Misses       1164     1177      +13     

Impacted Files	Coverage Δ
ctapipe/io/astropy_helpers.py	100.00% <100.00%> (ø)
ctapipe/io/tests/test_astropy_helpers.py	100.00% <100.00%> (ø)
ctapipe/instrument/guess.py	100.00% <0.00%> (ø)
ctapipe/instrument/tests/test_guessing.py	100.00% <0.00%> (ø)
ctapipe/instrument/tests/test_subarray.py	100.00% <0.00%> (ø)
ctapipe/io/tests/test_simteleventsource.py	100.00% <0.00%> (ø)
ctapipe/instrument/subarray.py	95.41% <0.00%> (+0.72%)	⬆️
ctapipe/io/datawriter.py	89.05% <0.00%> (+0.97%)	⬆️
ctapipe/io/simteleventsource.py	96.78% <0.00%> (+1.97%)	⬆️
ctapipe/io/dl1eventsource.py	94.27% <0.00%> (+3.66%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04d3f15...d4319d6. Read the comment docs.

[maxnoe]

do you know if the start/stop/step apply before or after the filtering? I was going to implement a chunked iterator wrapping read_table, so that may have an effect.

It's before, the rows are first selected based on the slice(start, stop, step) and then filtered by condition:

In [13]: f.root.test.read()
Out[13]: 
array([(0, -1.07400584), (1, -0.90504077), (2,  1.22623558),
       (3, -0.3141724 ), (4, -0.26661846), (5, -1.7715964 ),
       (6, -0.61327828), (7,  0.98540159), (8, -0.89481293),
       (9, -0.27684014)], dtype=[('idx', '<i8'), ('value', '<f8')])

In [17]: a[a['value'] < 0][::2]
Out[17]: 
array([(0, -1.07400584), (3, -0.3141724 ), (5, -1.7715964 ),
       (8, -0.89481293)], dtype=[('idx', '<i8'), ('value', '<f8')])

In [18]: f.root.test.read_where(step=2, condition='value < 0')
Out[18]: 
array([(0, -1.07400584), (4, -0.26661846), (6, -0.61327828),
       (8, -0.89481293)], dtype=[('idx', '<i8'), ('value', '<f8')])