allow YAML and TOML config files and multiple configs

[kosack]

fixes #1047
fixes #1732

This adds a function to load a YAML or TOML config file just like a JSON or Python one. There is more to do to make this fully useful, but it works as is.

It also adds the ability to specify multiple config files for a tool (by using repeated --config options, or a list in a config file itself).

For example the config file can look like:

DataWriter:
  Contact:
    # please fill in your contact information here. It will be stored in the
    # output files as provenance information
    name: YOUR-NAME-HERE
    email: [email protected]
    organization: YOUR-ORGANIZATION

  # options that control what is stored in the output file
  overwrite: false # do not overwrite existing files
  write_images: true # store DL1 images
  write_parameters: true # store DL1 parameters
  write_stereo_shower: false # store DL2 stereo geometry

CameraCalibrator:
  image_extractor_type: NeighborPeakWindowSum

ImageProcessor:
  image_cleaner_type: TailcutsImageCleaner

  # make sure you include a configuration for the image cleaner you selected
  # above here. The section named by the image_cleaner_type will be used to
  # configure it.
  TailcutsImageCleaner:
    # the thresholds for this image cleaner must be optimized for the data set
    # you are analyzing. The defaults may not be correct.
    picture_threshold_pe:
      - [type, "*", 10.0]
      - [type, LST_LST_LSTCam, 5.0]
      - [type, MST_MST_NectarCam, 5.0]
      - [type, SST_ASTRI_CHEC, 3.0]
    boundary_threshold_pe:
      - [type, "*", 5.0]
      - [type, LST_LST_LSTCam, 2.5]
      - [type, MST_MST_NectarCam, 2.5]
      - [type, SST_ASTRI_CHEC, 1.5]
    min_picture_neighbors:
      - [type, "*", 2]

  # These specify which images should be parameterized:
  ImageQualityQuery:
    quality_criteria:
      - [enough_pixels, "lambda im: np.count_nonzero(im) > 2"]
      - [enough_charge, "lambda im: im.sum() > 50"]

ShowerProcessor:
  # These specify criteria for telescopes that should be included in stereo
  # reconstruction:
  ShowerQualityQuery:
    quality_criteria:
      - [enough intensity, "lambda p: p.hillas.intensity > 50"]
      - [Positive width, "lambda p: p.hillas.width.value > 0"]
      - [enough pixels, "lambda p: p.morphology.num_pixels > 3"]

[maxnoe]

I am not such a huge fan of yml anymore... way too complex. I'd go for toml to have a simple config file format does not have the short comings of json / the unlimited power of py files

[codecov]

Codecov Report

Merging #1856 (ca66dba) into master (d254451) will increase coverage by 0.03%.
The diff coverage is 94.64%.

@@            Coverage Diff             @@
##           master    #1856      +/-   ##
==========================================
+ Coverage   92.00%   92.04%   +0.03%     
==========================================
  Files         189      189              
  Lines       14805    14842      +37     
==========================================
+ Hits        13622    13661      +39     
+ Misses       1183     1181       -2     

Impacted Files	Coverage Δ
ctapipe/core/tool.py	93.36% <90.62%> (+0.13%)	⬆️
ctapipe/tools/quickstart.py	77.58% <100.00%> (ø)
ctapipe/tools/tests/test_merge.py	97.18% <100.00%> (ø)
ctapipe/tools/tests/test_process.py	95.27% <100.00%> (+0.78%)	⬆️
ctapipe/io/datawriter.py	89.72% <0.00%> (+1.02%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d254451...ca66dba. Read the comment docs.

[kosack]

There is also an old function tool.export_tool_config_to_commented_yaml that could probably be cleaned up and resurrected once this PR is merged.

[kosack]

I am not such a huge fan of yml anymore... way too complex. I'd go for toml to have a simple config file format does not have the short comings of json / the unlimited power of py files

Could add TOML to this as well pretty easily. However, for nested configs, I really don't like TOML as it is harder to quickly see how it is nested. Simple YAML is pretty easy to read, I think, even if it does support some really crazy stuff. Anything is better than JSON, which is not really a configuration language, but rather just a serialization format (no comments, etc).

[kosack]

Now also supports TOML:

[DataWriter]
overwrite = false
write_images = true
write_parameters = true
write_stereo_shower = false
write_mono_shower = false
transform_image = true
transform_peak_time = true

[CameraCalibrator]
image_extractor_type = "NeighborPeakWindowSum"

[ImageProcessor]
image_cleaner_type = "TailcutsImageCleaner"

[DataWriter.Contact]
name = "YOUR-NAME-HERE"
email = "[email protected]"
organization = "YOUR-ORGANIZATION"

[ImageProcessor.TailcutsImageCleaner]
picture_threshold_pe = [ [ "type", "*", 10.0,], [ "type", "LST_LST_LSTCam", 5.0,], [ "type", "MST_MST_NectarCam", 5.0,], [ "type", "SST_ASTRI_CHEC", 3.0,],]
boundary_threshold_pe = [ [ "type", "*", 5.0,], [ "type", "LST_LST_LSTCam", 2.5,], [ "type", "MST_MST_NectarCam", 2.5,], [ "type", "SST_ASTRI_CHEC", 1.5,],]
min_picture_neighbors = [ [ "type", "*", 2,],]

[ImageProcessor.ImageQualityQuery]
quality_criteria = [ [ "enough_pixels", "lambda im: np.count_nonzero(im) > 2",], [ "enough_charge", "lambda im: im.sum() > 50",],]

[ShowerProcessor.ShowerQualityQuery]
quality_criteria = [ [ "enough intensity", "lambda p: p.hillas.intensity > 50",], [ "Positive width", "lambda p: p.hillas.width.value > 0",], [ "enough pixels", "lambda p: p.morphology.num_pixels > 3",],]

[kosack]

Probably there is a much nicer way to format the TOML version... I need to look at the spec. That was an auto-generated conversion from the JSON. I guess there are ways to wrap the list of lists we use for TelescopeConfigs and QualityCriteria

[kosack]

Any idea how to write nested lists on multiple lines correctly in TOML? I tried what I thought the manual said, and it doesn't work: This Fails to read:

[ImageProcessor.TailcutsImageCleaner]

picture_threshold_pe = [
    [ "type", "*", 10.0,],
    [ "type", "LST_LST_LSTCam", 5.0,],
    [ "type", "MST_MST_NectarCam", 5.0,],
    [ "type", "SST_ASTRI_CHEC", 3.0,],
]

Ah... TOML seems to be super picky about spaces after sections. Even Github won't render it

 toml.decoder.TomlDecodeError: Not a homogeneous array (line 28 column 1 char 634)

[kosack]

So TOML so far doesn't work with the TelescopeParameters...

1. It seems to require that all elements of an array are of the same format, so the nested arrays all have to be strings (then it stops complaining about not being able to parse the file
2. TelescopeParameter then of course fails to validate the value since it expects a float not a string.

Point 1 seems overly picky. Not sure if this is a problem with the implementation (I'm using toml not tomli), or the specification of TOML itself

[maxnoe]

TOML works fine, the toml library does not fully support toml 1.0. Using tomli works fine:

In [1]: import tomli

In [2]: !cat test.toml
[cuts]
foo = [
	["type", "*", 10.0],
	["type", "LST", 4.0],
	["id", 1, 4.0],
]

In [3]: tomli.load(open('./test.toml', "rb"))
Out[3]: {'cuts': {'foo': [['type', '*', 10.0], ['type', 'LST', 4.0], ['id', 1, 4.0]]}}

So I'd just try tomli and not toml

[kosack]

Yes tomli seems to work.

[kosack]

This reminds me one of the future PRs has to be to make these example files be generated automatically, including help text from the classes and trait options (similar to what traitlets.config does when it writes out sample python config files). For that, I will need to also add a way to specify which sections/classes to include in the example. Right now there are both the internal class and trait descriptions, and comments in the sample config files. That is unnecessary repetition, and things need to be updated in both places. I have to check if YAML or TOML lets you also include comments in the serialized output, or if it has to be done manually... I think not currently (except for some other implementations of yaml).

[kosack]

Will need to update the config files when #1875 is merged

[maxnoe]

Can you move the configs from ctapipe/tools/tests/resources to ctapipe/resources? They are not really only test resources anymore

[maxnoe]

LGTM