ECV Processors and Operations

When we currently say "processor", I can imagine at least three "API layers" to which this term could refer to. I describe them here because I think this is already a valid contribution to the architectural design to which we may adhere to when describing and implementing a new "processor":

• Cate (ESA CCI Toolbox) Low Level API Function(s): one or more Python functions that operate on the core CDM (common data model) or primitive parts of it. Such functions typically have the CDM (or a primitive part of it) as first argument and mostly use keyword arguments for more processing options. The functions implement most of the processor's logic and algorithms. Each function is described and specified in detail (extensive docstrings, see numpy docs as an example). Unit tests shall aim at covering 100% of their code.
• Cate CDM mixin: one or more methods that are added to the core CDM class that make calls to the low level functions, possibly with varying parameterisations. Mixins may be static (i.e. our CDM directly derives from a certain mixin class - typically for standard functionality), or dynamic, i.e. the CDM class is dynamically extended by some Python plugin module which simply assigns new methods to the CDM class - monkey patching it.
• Cate Processor (also "Operation"): The actual processor/operation implementations adhere to a common Cate Operation interface. They "know" about their parameter types, value ranges as well as their inputs and outputs. They can act as nodes in an acyclic processing graph. Users may create such graphs using JSON/XML files or using a graph builder tool (not in this version!). Processors are plugable software components that are registered in the Cate's global processor registry. When a processor is invoked (as part of a graph, a workflow, a command-line call, as an action from the desktop GUI), they make use of the corresponding low level Cate API Function or the (new) CDM mixin methods. The Cate frontend can dynamically create processor GUIs from the selected processor's input output descriptions. As an advanced example, you may look at the GPF (graph processing framework) and Operator interface in the SNAP toolbox:
• https://senbox.atlassian.net/wiki/display/SNAP/Sub-Systems+and+their+Extension+Points
• http://step.esa.int/docs/v2.0/apidoc/engine/org/esa/snap/core/gpf/Operator.html

It has currently gained even more attention due to its integration as a QGIS processing provider in the DHI GRAS software:

• https://github.com/DHI-GRAS/processing_gpf

The three layers described above could serve also as an implementation guideline. If we want to add a new "processor" functionality to the Cate, we create a new Python module and

1. write one or more functions (with extensive API documentation and unit tests)
2. write and expose a mixin class with the methods added to the Cate's CDM
3. write and expose a processor class that describes its inputs and outputs and makes calls to the functions or methods

Our own processor modules and also third party modules can "expose" their classes to Cate core via named plugin entry points "ect_cdm_mixin" and "ect_processor". See https://pythonhosted.org/setuptools/pkg_resources.html#entry-points. We do this in many other Python projects as well.