v0.7.0

What's Changed since v0.6.2

This is a major release with breaking API changes from the 0.6.x series. Scripts using ctapipe-0.6.x may need to be updated.

Notable API changes:

• gain-channel selection is applied at the DL1 level, so event.dl1.tel[i].image is now 1D (whereas before it was 2D, with the first index as the gain channel)
• removed Factory classes (factories now in constructor)
• EventSources are now constructed using EventSource.from_url() or EventSource.from_config(), thought the convenience function event_source() still works as before.
• The default EventSource for MC data is now the SimTelEventSource instead of HESSIOEventSource. This means libhessio is no longer required, as SimTelEventSource uses a pure-python implementation called pyeventio.
• Other EventSources are now plug-ins that are automatically detected if installed.
• the camera calibration code and ChargeExtractor have been totally re-factored and simplified. Note that it's lower-level API has thus changed.
• ctapipe-extra is not longer strictly required to be installed (though some functionality will not work if it is not installed like constructing a CameraGeometry from name if no CTAPIPE_SVC_PATH is set by the user).
• New EngineeringCameraFrame coordinate system that allows one to draw a camera as it would look when parked (rather than the standard SimTelArray coordinate system).
• naming scheme for telescopes has changed in the instrument module (it is now <optics type>_<optics_subtype>_<camera type>, e.g. LST_LST_LSTCam or SST_ASTRI_CHEC)

Improvements:

• ctapipe no longer needs any compilation (the c code in CameraCalibrator has been removed and replaced with Numba functionality for speed and maintainability improvements), which means the package can be made architecture-independent (though python must generate the *.pyc files on install, which takes a bit longer)
• HDF5TableWriter now can write tables to any group(s) in an HDF5 file, and can optionally append prefixes to the output columns of a Container (e.g. width → hillas_width)
• HillasReconstructor now supports non-parallel (e.g. divergent or convergent) pointings
• many speed improvements for calibration and neighbor matrix operations.
• New tailcuts (picture/boundary) function for multiple nearest-neighbor cleaning (MARS-like)

Detailed Changelog

• Clean up setup.py (#1118) @kosack
• remove unused dependencies in environment.yml (#1113) @vuillaut
• Reduce numba signatures to improve import time (#1108) @watsonjj
• Remove unused utils (#1112) @maxnoe
• Add method to read bright star catalog and find bright stars in sky region (#1105) @moritzhuetten
• Add Fields to calibration containers (#1111) @FrancaCassol
• Reconstruction fix (also for Divergent Pointing) (#946) @thomasgas
• updated AUTHORS with latest mailmap (#1107) @kosack
• Restore apply_time_delta_cleaning as a public method (#1106) @watsonjj
• Update mailmap (#1103) @maxnoe
• Add uncertainties to TimingParemeters (#1101) @maxnoe
• Close simtel eventsource in exit (#1104) @maxnoe
• Delete .coverage.dapmcw186.24220.071001 (#1099) @cdeil
• First version of the MARS tailcut (1st pass) (#1096) @HealthyPear
• Add gain selection to calibration chain (#1095) @watsonjj
• Added my emails to mailmap (#1098) @HealthyPear
• Restore returning selected channels in gain selection (#1094) @watsonjj
• Refactoring of GainSelector (#1093) @watsonjj
• Integer hdftable (#1089) @maxnoe
• Add EngineeringCameraFrame (#1085) @maxnoe
• Coordinates notebook pr (#1075) @maxnoe
• Fix HillasReconstructor in case of too few telescopes (#994) @LukasNickel
• Fix writing of booleans, fixes #1082 (#1083) @maxnoe
• Add deviation parameter to timing_parameters (#1079) @watsonjj
• Fix broken telescope id string, Fixes #1058 (#1069) @kosack
• Improvements to Tool and Component functionality (#1052) @kosack
• Allow tables in arbitrarily deep hdf5 groups (#1068) @kosack
• Set python_requires in setup.py (#1074) @maxnoe
• Camera calibration code (#1067) @FrancaCassol
• adding support for int8 in PYTABLES_TYPE_MAP (#1072) @vuillaut
• normalize index column name in subarray table (#1065) @kosack
• Fix HDF5TableWriter to ensure user-defined col transforms are applied (#1060) @kosack
• Update eventio to 0.20 (#1062) @maxnoe
• Fix number of mirrors, fixes #1057 (#1061) @maxnoe
• Consolidate camera calibration in a single class (#1048) @watsonjj
• Use sparse matrix to calculate neighbor_matrix_where (#1056) @maxnoe
• Cache camera geometries by camera name (#1054) @maxnoe
• Suggestion: Retire CameraR1Calibrator (#1046) @watsonjj
• Create numba ufunc for sum of samples within charge extraction window (#1038) @watsonjj
• Implement nan-handling like matplotlib high-level api (#1050) @maxnoe
• Fix See Also docs for sphinx 2 (#1051) @maxnoe
• Create improved algorithm for pulse time (#1045) @watsonjj
• Update bokeh plotters to handle nan (#1039) @watsonjj
• Fix passing config to CameraCalibrator (#1042) @watsonjj
• Replace WaveformCleaner and ChargeExtractor with WaveformExtractor (#1033) @watsonjj
• Only detect plugins when they need to exist in global (#1036) @watsonjj
• Make ctapipe-extra truly optional (#1002) @maxnoe
• Fix for TargetIOR1Calibrator (#1035) @watsonjj
• Replace C function get_sum_array with a Numba implementation (#1032) @watsonjj
• ChargeExtractor tests and simplification (#1023) @watsonjj
• allow enums in containers and support in tableio (#978) @dneise
• Add tests to Tools to check that help message works (#1034) @watsonjj
• Fix neighbors (#1015) @maxnoe
• Fix component docs (#1016) @maxnoe
• Remove targetio event source and calibrator (#1012) @maxnoe
• remove sst1m_event_source (#1011) @dneise
• Update requirements (#1008) @maxnoe
• Use conda packages instead of pip (#989) @kosack
• Fix traitlets config (#998) @maxnoe
• Use new conda syntax for activate (#999) @maxnoe
• Set defaults only for fields not passed in init (#1001) @maxnoe
• Remove LSTEventsource, (#992) @maxnoe
• Remove nectarcam event source, now in ctapipe_io_nectarcam (#993) @maxnoe
• Refactor toymodel, add SkewedGaussian (#996) @maxnoe
• Refactor instrument module (#982) @maxnoe
• fixed bug in coordinate transformation (#997) @seinecke
• Pass allowed_tels down to eventio in simtel event source (#987) @maxnoe
• Add missing imports for event sources (#991) @maxnoe
• Waveform cleaner (#973) @FrancaCassol
• Moved a bunch of examples to tools (#955) @kosack
• Use eventio v0.17.1 (#966) @dneise
• Remove HorizonFrame and use AltAz directly from astropy, fixes #949 (#975) @maxnoe
• Move get_shower_coordinates to hillas, fixes #934 (#976) @maxnoe
• Fix set_line_hillas (#974) @kpfrang
• Alternative Take on Factories (#916) @dneise
• Passing masked array to hillas_parameters (#969) @watsonjj
• make pyhessio not necessary for normal users (#957) @dneise
• Do not call everything stable (#956) @maxnoe
• Avoid underscore replacement empty prefixes. (#951) @kbruegge
• Improve Component argument consistency (#947) @watsonjj

v0.6.2

Algorithmic Changes:

• Fixes major bug in impact-parameter reconstruction where an incorrect value was calculated in the case of diffuse simulations

What's Changed since v0.6.1

• Fix unknown section example warning (#800) @maxnoe
• fixed ranges of cam-display so they correspond to fixed toymodel sims (#808) @kosack
• Implement number of islands (#801) @LukasNickel
• fix package name for yaml (should be pyyaml) (#810) @kosack
• fix ctapipe build failure (#811) @kosack
• Add test that shows slicing breaks cam geom and fix it (#782) @maxnoe
• freeze ctapipe-extra to version 0.2.14 since 0.2.15 has a bug (#815) @kosack
• Bokeh plotter (#714) @watsonjj
• scaling factor in muon_integrator (#774) @AMWMitchell
• quick fix of example coordinate_transformation (#816) @thomasarmstrong
• Modifying check_calib.ipynb to work with the new CameraCalibrator API (#814) @vuillaut
• Fix impact reco, fixes #828, #829 (#830) @maxnoe
• First version of notebook for LST analysis bootcamp (#825) @maxnoe
• Fix warning about already closed hessio file (#834) @maxnoe
• fix errordef > 0. (#839) @rlopezcoto
• Add test how it should be (#842) @maxnoe
• Use sparse neighbor matrix (#826) @maxnoe
• SimTelEventSource using pyeventio (#864) @maxnoe
• Fix subarray peek. No more warnings. (#841) @thomasgas
• Add .mailmap (#894) @maxnoe
• Add scikit learn to install requires (#877) @dneise
• Additional metadata from pyhessio (#895) @dneise
• Cleaning the ctapipe folder. (#892) @thomasgas
• Remove flow submodule (got moved to its own package) (#893) @maxnoe
• remove config=None, tool=None in some tests (#897) @dneise
• Implemented FACT image cleaning (#862) @LukasNickel
• Updates of NectarCam and LSTCam real data eventsource class (#812) @FrancaCassol
• Fix a few test warnings (#902) @maxnoe
• Fix SST-1M to be DC and not SC. Solves #905 (#908) @thomasgas
• Import bokeh.palletes correctly, fixes #911 (#912) @maxnoe
• Do not set autodownload, fixes doc build (#910) @maxnoe
• Fix docstring of EventSeeker, fixes #768 (#914) @maxnoe
• Remove all hillas_paramters but version 5, fixes #866 (#904) @maxnoe
• deleted summary.html - accidentally committed file (#919) @kosack
• Remove serializer, fixes #887 (#913) @maxnoe
• Charge Resolution Update (#827) @watsonjj
• remove optional deps in order to fix tests (#925) @dneise
• Prefix containers (#833) @maxnoe
• speed up is_compatible; makes event_source factory faster (#927) @dneise
• Major coordinate refactoring, fixes #899, #900, #505, #416 (#896) @maxnoe
• Print duration of notebooks on travis (#928) @dneise
• clean up environment.yml (#932) @kosack
• cleanup examples/notebooks to increase test speed (#933) @kosack
• fix pandas to 0.23.4 for the time being (#937) @dneise
• fix bug where wrong value was filled for num_mirror_tiles (#939) @kosack
• Add additional tests to Component (#935) @watsonjj
• Add get_pixel_id method (#918) @moritzhuetten
• fixed examples that use pandas to read hdf5 (#940) @kosack
• Example notebooks in Sphinx documentation (#941) @kosack
• Correct filling of photo_electron_image container in SimtelEventSource (#943) @watsonjj
• replace "make develop" with "pip install -e ." (#945) @dneise
• Added additional MC header information (#921) @LukasNickel
• Rename average_size to average_intensity (#953) @mackaiver
• Filter HorizonFrame warnings during reconstruction (#954) @mackaiver
• modernize formatting code for python 3.6+ (#944) @kosack

v0.6.1

What's Changed since v0.6.0

• Fix broken build (#743) @kosack
• Add example script for a simple event writer (#746) @jjlk
• Fix camera axis alignment in HillasReconstructor (#741) @mackaiver
• Lst reader (#749) @FrancaCassol
• replace deprecated astropy broadcast (#754) @mackaiver
• A few more example notebooks (#757) @kosack
• Add MC xmax info (#759) @mackaiver
• Use Astropy Coordinate Transofmations For Reconstruction (#758) @mackaiver
• Trigger pixel reader (#745) @thomasarmstrong
• Change requested in #742: init Hillas skewness and kurtosis to NaN (#744) @STSpencer
• Fix call to np.linalg.leastsq (#760) @kosack
• Fix/muon bugs (#762) @kosack
• Implement hillas features usen eigh (#748) @maxnoe
• Use HillasParametersContainer only (#763) @maxnoe
• Regression features in RegressorClassifierBase (#764) @vuillaut
• Adding an example notebook no how to convert hex geometry to square and back (#767) @vuillaut
• Wrong angle in ArrayDisplay. changed phi to psi. (#771) @thomasgas
• Unstructured interpolator (#770) @ParsonsRD
• Lst reader (#776) @FrancaCassol
• Fixing core reconstruction (#777) @kpfrang
• Leakage (#783) @maxnoe
• Revert "Fixing core reconstruction" (#789) @kosack
• Fixing the toy image generator (#790) @maxnoe
• Fix bad builds by changing channel name (missing pyqt package) (#793) @kosack
• Implement concentration image features (#791) @maxnoe
• updated main documentation page (#792) @kosack
• Impact intersection (#778) @mackaiver
• add test for sliced geometries for hillas calculation (#781) @mackaiver
• Simple HESS adaptations (#794) @ParsonsRD
• added a config file for github release-drafter plugin (#795) @kosack
• Array plotting (#784) @thomasgas
• Minor changes: mostly deprecationwarning fixes (#787) @mireianievas
• Codacy code style improvements (#796) @dneise
• Add unit to h_max in HillasReconstructor (#797) @jjlk
• speed up unit tests that use test_event fixture (#798) @kosack
• Update Timing Parameters (#799) @LukasNickel

v0.6.0

This is an interim release, after some major refactoring, and before we add the automatic gain selection and refactored container classes. It's not intended yet for production.

Some Major changes since last release:

• new EventSource class hierarchy for reading event data, which now supports simulation and testbench data from multiple camera prototypes (notably CHEC, SST-1M, NectarCam)
• new EventSeeker class for (inefficient) random event access.
• a much improved Factory class
• re-organized event data structure (still evolving) - all scripts not in ctapipe must be changed to work with the new data items that were re-named (a migration guide will be given in the 0.7 release)
• better HDF5 table output, supporting merging multiple Containers into a single output table
• improvements to Muon analysis, and the muon example script
• improvements to the calibration classes
• big improvements to the Instrument classes
• lots of cleanups and bug fixes
• much more...

v0.5.2

release version 0.5.2

instrument restructuring

• instrument module now has much more rich functionality (SubarrayDescription, TelescopeDescription, OpticsDescription classes added
• no more need to construct CameraGeometry and others manually, they are created in the hessio_event_source
• all new code should use event.inst.subarray
• the old inst.tel_pos, inst.optics_foclen, etc, will be phased out in the next point release (but still exist in this release)

ready for sensitivity plots

Lots of improvements and bug fixes, some highlights listed below.

Major framework changes:

• removed astropy_helpers and ctapipe_extras sub-repos (the latter is now a seprate module)
• improved setup, versioning, and package system
• improved documentation
• working unit tests
• support for travisCI documentation builds
• simplified instrument module
• faster and more unified CameraGeometry, with support for cameras in FITS files
• basic provenance system to record input/output/parameter metadata
• improved coordinate systems
• event-wise HDF5 table io
• improved camera and array displays
• ctapipe-dump-instrument tool

Major algorithm changes:

• complete Muon analysis
• complete Hillas reconstruction (HillasReconstructor)
• energy estimation with machine learning
• ImPACT reconstruction and fast table interpolation
• cleaned up ChargeExtractors and WaveFormCleaners
• simplified calibration (and added CameraCalibrator helper)

v0.4.0

API Changes

• New Container structure that follows CTA data levels better

All Changes:

0.4.0 / 2017-03-13

• updated conda environment.yml and setup instructions to use it
• add History.md
• update notebook
• add example provenance notebook
• add Provenance.clear() method
• revert .9 -> .1
• forgot adding this file to staging
• removed rotation of astri cameras
• typo
• moved uncertainty from predict to fit_core_crosses
• add automatic provenance in Tool and io
• added uncertainty on position fit
• fixed missing camera rotation
• fix bug in ctapipe-dump-triggers tool
• use Singleton metaclass for provenance
• refactoring
• separate activity provenance from global provenance
• cleanups to proveance class and core init
• update provenence
• mini change in doc string
• first attempt at system provenance and time sampling
• Corrected tests for charge_resolution
• Corrected example cmdline arguments
• some updates to example notebooks
• Corrected r0 to dl0
• merge and apply fixes to PR#284
• Replaced event.dl0 calls to event.r0
• small fix to example
• add --all option to ctapipe-info
• fix broken --dependencies option in ctapipe-info
• fix in camera_rotation
• fix bug in CameraDisplay
• mask in geometry now uses bools instead of int
• minor camera display update
• use cta-observatory conda channel for pyhessio install
• remove version cache from git repo and make sure it's in .gitignore
• improve camdemo tool
• add version cache to gitignore
• remove import of unused npLinAlgError
• replaced least chi2 calculation with numpy build-in
• stuff
• Added pedestal_path and tf_path to CameraR1CalibratorFactor, as real cameras require these separate files for calibration
• Corrected bug in Factory
• When saving the pickle file, the directory will created if it doesn't exist
• Some refactoring
• Update Charge Resolution
• Small change to how integration window is plotted
• Included extra external children
• Corrected container
• Corrected @observe logging
• Corrected container for DL0
• Updated examples/display_integrator.py to use new calibration steps
• Updated calibration pipeline to use new calibration steps
• Renamed examples/dl1_calibration.py back to examples/calibration_pipeline.py as it now contains the whole calibration chain from r0 to dl1
• Corrections to pass tests
• Added tests for calibrators
• Removed calibrate_source
• Corrected check_*_exists
• Renamed test files Renamed MCR1Calibrator to HessioR1Calibrator
• Created function check_*_exists in each calibration step to allow them to be ran even if the data has been read at a later calibration step
• Created dl0.py to handle to data volume reduction in the conversion from r1 to dl0. Created reductors.py to contain the date volume reductors that can be passed as an argument to CameraDL0Reducer in dl0.py.
• Corrected r1.py to loop through all telescopes that have data in the event
• Removed dl0 correction for mc files - this is now handled by r1.py
• Removed mc.py and mycam.py as they do not fit in the new calib methodology
• Created r1 calibrator component - should replace mc.py
• Added origin attribute to EventFileReader class - useful for components that depend on the file type
• Updated docstring
• Renamed ctapipe/calib/camera/calibrators.py to ctapipe/calib/camera/dl1.py
• Removed clearing of dl2
• Imported external classes that should be included in the factory
• Refactored dl0 container to r0 container Created new containers: r1 (containing r1 calibrated data) and dl0 (containing data volume reduced data)

v0.3.4

Merge pull request #323 from kosack/master

more fixes for version.py

new package

Merge pull request #316 from kosack/master

fix build system so packages can be made