Merge pull request #384 from sartography/improvement/documentation-re…

…write

make documentation more developer-friendly

RELEASE_NOTES.md

@@ -6,9 +6,11 @@ added a few new features to support [SpiffArena](https://www.spiffworkflow.org/)
 ## Breaking Changes from 2.x:
 
 * We refactored task iteration to allow for greater flexibility in selecting tasks.  Usage of `workflow.get_tasks` has changed.
+  See [Filtering Tasks](https://spiffworkflow.readthedocs.io/en/latest/bpmn/workflows.html#filtering-tasks) for documentation.
 * We've continued to reorganize files in order to impose more structure on the package.  Imports have changed (but we've added
-  `__init__.py` files to make these reorganizations less painful in the future).
-* We've simplified the BPMN serializer.
+  `__init__.py` files to make these reorganizations less painful in the future).  See an overview of the package structure at
+  [What's in the BPMN Module](https:://spiffworkflow.readthedocs.io/en/latest/bpmn/imports.html).
+* We've simplified the BPMN serializer. [BPMN Serializer](https:://spiffworkflow.readthedocs.io/en/latest/bpmn/serialization.html)
 
 ## Features and Improvements
 
@@ -50,4 +52,4 @@ added a few new features to support [SpiffArena](https://www.spiffworkflow.org/)
 * Removed dependencies on descendant tasks from parallel gateway merges that could cause workflow execution to stall.
 * Ensured that all dependencies of a subprocess are identified by the BPMN parser.
 
-**Full Changelog**: https://github.com/sartography/SpiffWorkflow/compare/v2.0.1...main
+**Full Changelog**: https://github.com/sartography/SpiffWorkflow/compare/v2.0.1...v3.0.0rc0

doc/bpmn/application.rst

@@ -0,0 +1,163 @@
+Overview
+========
+
+This section focuses on the example application rather than the library itself; it is intended to orient people
+attempting to use this documentation, so we won't devote that much space to it (much of it is necessary for a 
+functioning app, but not directly relevant to library use); nonetheless, there is quite a bit of code and a general
+idea of what's here will be helpful.
+
+The application has several parts:
+
+- an engine, which uses SpiffWorkflow to run, parse, and serialize workflows
+- a curses UI for running and examining Workflows, which uses the engine
+- a command line UI with some limited functionality, which also uses the engine
+
+We'll mainly focus on the engine, as it contains the interface with the library, though a few examples will come from
+the other components.  The engine is quite small and simple compared to the code required to handle user input and
+display information in a terminal.
+
+Configuration is set up in a python module and passed into the application with the `-e` argument, which loads the
+configured engine from this file.  This setup should make it relatively to change the behavior of engine.  The
+following configurations are included:
+
+- :code:`spiff_example.spiff.file`: uses spiff BPMN extensions and serializes to JSON files
+- :code:`spiff_example.spiff.sqlite`: uses spiff BPMN extensions and serializes to SQLite
+- :code:`spiff_example.camunda.default`: uses Camunda extensions and serializes to SQLite
+
+.. _quickstart:
+
+Quickstart
+==========
+
+There are several versions of a product ordering process of variying complexity located in the
+:example:`bpmn/tutorial` directory of the repo which contain most of the elements that SpiffWorkflow supports.  These
+diagrams can be viewed in any BPMN editor, but many of them have custom extensions created with
+`bpmn-js-spiffworflow <https://github.com/sartography/bpmn-js-spiffworkflow>`_.
+
+To add a workflow via the command line and store serialized specs in JSON files:
+
+.. code-block:: console
+
+   ./runner.py -e spiff_example.spiff.file add \
+      -p order_product \
+      -b bpmn/tutorial/{top_level,call_activity}.bpmn \
+      -d bpmn/tutorial/{product_prices,shipping_costs}.dmn
+
+To run the curses application using serialized JSON files:
+
+.. code-block:: console
+
+   ./runner.py -e spiff_example.spiff.file
+
+Select the 'Start Workflow' screen and start the process.
+
+The Application in Slightly More Depth
+======================================
+
+The application requires the name of a module to load that contains a configuration such as one of those defined above.
+
+To start the curses UI using the JSON file serializer:
+
+.. code-block:: console
+
+   ./runner.py -e spiff_example.spiff.file
+
+If the application is run with no other arguments, the curses UI will be loaded.
+
+It is possible to add a workflow spec through the curses UI, but it is going to be somewhat painful to do so unless
+you are a better typist and proofreader than I; therefore, there are also a few command line utilities for handling
+some of the functionality, including adding workflow specs.
+
+Command line options are
+
+- :code:`add` to add a workflow spec (while taking advantage of your shell's file completion functionality)
+- :code:`list` to list the available specs
+- :code:`run` to run a workflow non-interactively
+
+Each of these options has a help menu that describes how to use them.
+
+Configuration Modules
+=====================
+
+The three main ways that users can customize the library are:
+
+- the parser
+- the script engine
+- the serializer
+
+We use the configuration module to allow these components to be defined outside the workflow engine itself and passed
+in as parameters to make it easier to experiment. I am somewhat regularly asked questions about why a diagram doesn't
+executed as expected, or how to get the script engine to work a particular way; this is a first pass at setting
+something up that works better for me than configuring the library's test loader and running that in a debugger; I hope
+other people will find it useful as well.
+
+We'll go through the configuration in greater detail in later sections, but we'll take a brief look at the simplest
+configuration, :app:`spiff/file.py` here.
+
+In this file, we'll initialize our parser:
+
+.. code-block:: python
+
+    parser = SpiffBpmnParser()
+
+We don't need to further customize this parser -- this is a builtin parser that can handle DMN files as well as Spiff
+BPMN extensions.
+
+We also need to initialize a serializer:
+
+.. code-block:: python
+
+    dirname = 'wfdata'
+    FileSerializer.initialize(dirname)
+    registry = FileSerializer.configure(SPIFF_CONFIG)
+    serializer = FileSerializer(dirname, registry=registry)
+
+JSON specs and workflows will be stored in :code:`wfdata`.  The :code:`registry` is the place where information about
+converting Python objects to and from JSON-serializable dictionary form is maintained.  :code:`SPIFF_CONFIG` tells the
+serializer how to handle objects used internally by Spiff.  Workflows can also contain arbitrary data, so this registry
+can also tell the serializer how to handle any non-serializable data in your workflow.  We'll go over this in more
+detail in :ref:`serializing_custom_objects`.
+
+We initialize a scripting enviroment:
+
+.. code-block:: python
+
+    script_env = TaskDataEnvironment({'datetime': datetime })
+    >script_engine = PythonScriptEngine(script_env)
+
+The :code:`PythonScriptEngine` handles execution of script tasks and evaluation of gateway and DMN conditions.
+We'll create the script engine based on it; execution and evaluation will occur in the context of this enviroment.
+
+SpiffWorkflow provides a default scripting environment that is suitable for simple applications, but a serious
+application will probably need to extend (or restrict) it in some way.  See :doc:`script_engine` for a few examples.
+Therefore, we have the ability to optionally pass one in.
+
+In this case, we'll include access to the :code:`datetime` module, because we'll use it in several of our script tasks.
+
+We also specify some handlers:
+
+.. code-block:: python
+
+    handlers = {
+        UserTask: UserTaskHandler,
+        ManualTask: ManualTaskHandler,
+        NoneTask: ManualTaskHandler,
+    }
+
+This is a mapping of task spec to task handler and lets our application know how to handle these tasks.
+
+.. note::
+
+    In our application, we're also passing in handlers, but this is not a typical use case.  The library knows how to
+    handle all task types except for human (User and Manual) tasks, and those handlers would typically be built into
+    your application.  However, this application needs to be able to deal with more than one set of human task specs,
+    and this is a convenient way to do this.  The library treats None tasks (tasks with no specific type assigned)
+    like Manual Tasks by default.
+
+We then create our BPMN engine (:app:`engine/engine.py`) using each of these components:
+
+.. code-block:: python
+
+    from ..engine import BpmnEngine
+    engine = BpmnEngine(parser, serializer, handlers, script_env)
+

doc/bpmn/camunda/index.rst

@@ -0,0 +1,131 @@
+Using the Camunda Configuration Module
+======================================
+
+.. warning:: There is a better way ...
+  SpiffWorkflow does not aim to support all of Camunda's proprietary extensions.
+  Many of of the items in the Camunda Properties Panel do not work.  And
+  major features of SpiffWorkflow (Messages, Data Objects, Service Tasks, Pre-Scripts, etc...)
+  can not be configured in the Camunda editor.  Use `SpiffArena <https://www.spiffworkflow.org/posts/articles/get_started/>`_
+  to build and test your BPMN models instead!
+
+Earlier users of SpiffWorkflow relied heavily on Camunda's modeler and several of our task spec
+implementations were based on Camunda's extensions.  Support for these extensions has been moved
+to the :code:`camunda` package.  We are not actively maintaining this package (though we will
+accept contributions from Camunda users!).  Please be aware that many of the Camunda extensions
+that will appear in the Camunda editor do not work with SpiffWorkflow.
+
+In this repo, we provide the following configuration:
+
+.. code-block:: console
+
+   ./runner.py -e spiff_example.camunda.sqlite
+
+Tasks
+=====
+
+User Tasks
+----------
+
+Creating a User Task
+^^^^^^^^^^^^^^^^^^^^
+
+When you click on a user task in the BPMN modeler, the Properties Panel includes a form tab. Use this
+tab to build your questions.
+
+The following example shows how a form might be set up in Camumda.
+
+.. figure:: figures/user_task.png
+   :scale: 30%
+   :align: center
+
+   User Task configuration
+
+
+Manual Tasks
+------------
+
+Creating a Manual Task
+^^^^^^^^^^^^^^^^^^^^^^
+
+We can use the BPMN element Documentation field to display more information about the context of the item.
+
+Spiff is set up in a way that you could use any templating library you want, but we have used 
+`Jinja <https://jinja.palletsprojects.com/en/3.0.x/>`_.
+
+In this example, we'll present an order summary to our customer.
+
+.. figure:: figures/documentation.png
+   :scale: 30%
+   :align: center
+
+   Element Documentation
+
+Example Code
+------------
+
+Example Human task handlers can be found in :app:`camunda/curses_handlers.py`.
+
+Events
+======
+
+Message Events
+--------------
+
+Configuring Message Events
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. figure:: figures/throw_message_event.png
+   :scale: 60%
+   :align: center
+
+   Throw Message Event configuration
+
+
+.. figure:: figures/message_start_event.png
+   :scale: 60%
+   :align: center
+
+   Message Catch Event configuration
+
+The Throw Message Event Implementation should be 'Expression' and the Expression should
+be a Python statement that can be evaluated.  In this example, we'll just send the contents
+of the :code:`reason_delayed` variable, which contains the response from the 'Investigate Delay'
+Task.
+
+We can provide a name for the result variable, but I have not done that here, as it does not
+make sense to me for the generator of the event to tell the handler what to call the value.
+If you *do* specify a result variable, the message payload (the expression evaluated in the
+context of the Throwing task) will be added to the handling task's data in a variable of that
+name; if you leave it blank, SpiffWorkflow will create a variable of the form <Handling
+Task Name>_Response.
+
+MultiInstance Tasks
+===================
+
+Earlier versions of SpiffWorkflow relied on the properties available in the Camunda MultiInstance Panel.
+
+.. figure:: figures/multiinstance_task_configuration.png
+   :scale: 60%
+   :align: center
+
+   MultiInstance Task configuration
+
+SpiffWorkflow has a MultiInstance Task spec in the :code:`camunda` package that interprets these fields
+in the following way:
+
+* Loop Cardinality:
+
+   - If this is an integer, or a variable that evaluates to an integer, this number would be 
+     used to determine the number of instances
+   - If this is a collection, the size of the collection would be used to determine the number of
+     instances
+
+* Collection: the output collection (input collections have to be specified in the "Cardinality" field).
+
+* Element variable: the name of the varible to copy the item into for each instance.
+
+.. warning::
+
+   The spec in this package is based on an old version of Camunda, so the panel may have changed.  The
+   properties might or might not have been the way Camunda used these fields, and may or may not be similar
+   to newer or current versions.  *Use at your own risk!*