Various Colang 2 language reference documentation updates

docs/colang_2/getting_started/index.rst

@@ -22,7 +22,10 @@ The ``config.yml`` file for all the examples should have the following content:
       engine: openai
       model: gpt-3.5-turbo-instruct
 
-The above config sets the Colang version to "2.x" (this is needed since "1.0" is currently the default) and the LLM engine to OpenAI's ``gpt-3.5-turbo-instruct``.
+The above config sets the Colang version to "2.x" (this is needed since "1.0" is currently the default) and the LLM engine to OpenAI's ``gpt-3.5-turbo-instruct``. Make sure to set the required API access key as a environment variable (e.g. OPENAI_API_KEY for OpenAI API).
+
+.. note::
+  Check the section :ref:`development-and-debugging` for how you can install Colang syntax highlighting to make editing Colang scripts easier.
 
 Terminology
 -----------

docs/colang_2/getting_started/llm-flows.rst

@@ -8,7 +8,7 @@ This section explains how to create LLM-driven flows in Colang 2.0.
 
 Using Colang, you can describe complex patterns of interaction. However, as a developer, you will never be able to describe all the potential paths an interaction can take. And this is where an LLM can help, by generating *LLM-driven continuations* at runtime.
 
-The :ref:`colang_2_getting_started_dialog_rails` and the :ref:`colang_2_getting_started_input_rails` examples, show how to use the LLM to generate continuations dynamically. The example below is similar to the dialog rails example, but it instructs to LLM to generate directly the bot response:
+The :ref:`colang_2_getting_started_dialog_rails` and the :ref:`colang_2_getting_started_input_rails` examples, show how to use the LLM to generate continuations dynamically. The example below is similar to the dialog rails example, but it instructs to LLM to generate directly the bot response. Note, the quality of the response depends on the configured LLM model and can vary.
 
 
 .. code-block:: colang

docs/colang_2/language_reference/defining-flows.rst

@@ -15,7 +15,7 @@ Defining Flows
 Introduction
 ----------------------------------------
 
-So far you have seen only one flow, the main flow. But in Colang we can define many different flows, like functions in other programming languages. A flow defines a specific interaction pattern made of a sequence of statements. It has a name that can contain whitespace characters and has optional in and out parameters with optional default values.
+So far you have seen only one flow, the main flow. But in Colang we can define many different flows, like functions in other programming languages. A flow defines a specific interaction pattern made of a sequence of statements. It has a name made of lowercase letters, numbers, underline and whitespace characters. Additionally, it can define in and out parameters with optional default values.
 
 .. important::
     Flow syntax definition:
@@ -42,6 +42,11 @@ So far you have seen only one flow, the main flow. But in Colang we can define m
             """User said something."""
             # ...
 
+    The choice of allowing whitespace characters in flow names comes with some limitations:
+
+    * The keywords ``and``, ``or`` and ``as`` cannot be used in flow names and would need to be escaped with a leading underline character. But often, rather than using e.g. the word 'and', you can use the word 'then' to combine to actions, e.g ``bot greet then smile`` to describe the sequential dependency. Or alternatively write it as ``bot greet smiling`` if it happens concurrently.
+    * As shown in chapter :ref:`Working with Variables & Expressions <working-with-variables-and-expressions>` variables will always start with a ``$`` character.
+
 Like an action, a flow can be started and waited for to finish using the keywords ``start``, ``await`` and ``match``:
 
 .. code-block:: colang
@@ -82,7 +87,12 @@ Note, that starting a flow will immediately process and trigger all initial stat
 .. important::
     Starting a flow will immediately process and trigger all initial statements of the flow, up to the first statement that waits for an event.
 
-Similar to an action, flows themselves can generate different events which have priority over other events (see :ref:`Internal Events<internal-events-defining-flows>`):
+
+------------
+Flow events
+------------
+
+Similar to actions, flows themselves can generate different events which have priority over other events (see :ref:`Internal Events<internal-events-defining-flows>`):
 
 .. code-block:: colang
 
@@ -123,7 +133,7 @@ Here is an example of a flow with parameters:
     flow bot say $text $volume=1.0
         await UtteranceBotAction(script=$text, intensity=$volume)
 
-Note how we can abstract and simplify the action handling with flows using a simpler name. This allows us to wrap most actions and events into flows that are made readily available through the :ref:`the-standard-library`.
+Note how we can abstract and simplify the action handling with flows using a simpler name. This allows us to wrap most actions and events into flows that are made readily available through the :ref:`the-standard-library`. See also section :ref:`Internal Events <internal-events-defining-flows>` where the underlying flow event mechanics are explained in more detail.
 
 ----------------------------------------
 Flow and Action Lifetime
@@ -570,8 +580,6 @@ You might have spotted by now the deliberate use of tenses in the naming of flow
 - Use the form ``<subject> started <verb continuous form> ...`` to describe an action that has started, e.g. ``bot started saying something`` or ``user started saying something``
 - Start with the noun or gerund form of an activity for flows that should be activated and that wait for a certain interaction pattern to react to, e.g. ``reaction to user greeting``, ``handling user leaving`` or ``tracking bot talking state``.
 
-Since flow names allow whitespace characters and we have the grouping keywords ``and`` and ``or``, flow names can currently not contain these two keywords as part of their name. Often, rather than using the word 'and' you can use the word 'then' to combine to actions, e.g ``bot greet then smile`` to describe the sequential dependency. Or write it as ``bot greet smiling`` if it happens concurrently.
-
 
 .. _action-like-and-intent-like-flows:
 

docs/colang_2/language_reference/make-use-of-llms.rst

@@ -43,7 +43,12 @@ One of the main LLM generation mechanism in Colang are the so-called Natural Lan
     # Use an existing variable in NLD
     $response_to_user = ..."Provide a brief summary of the current order. Order Information: '{$order_information}'"
 
-Every NLD will be interpreted and replaced during runtime by the configured LLM backend and can be used in Colang to generate context dependent values. Alternatively, you can also describe the purpose and function of a flow using a docstring like NLD at the beginning of a flow. Using a standalone generation operator in the flow will use the flows NLD to infer the right flow expansion automatically:
+Every NLD will be interpreted and replaced during runtime by the configured LLM backend and can be used in Colang to generate context dependent values. With NLDs you are able to extract values and summarize content from the conversation with the user or based on results from other sources (like a database or an external service).
+
+.. note::
+    NLDs together with the variable name are interpreted by the LLM directly. Depending on the LLM you use you need to make sure to be very specific in what value you would like to generate. It is good practice to always clearly specify how you want the response to be formatted and what type it should have (e.g., ``$user_name = ..."Return the user name as single string between quotes''. If no user name is available return 'friend'"``.
+
+Alternatively, you can also describe the purpose and function of a flow using a docstring like NLD at the beginning of a flow. Using a standalone generation operator in the flow will use the flows NLD to infer the right flow expansion automatically:
 
 .. code-block:: colang
 

docs/colang_2/language_reference/more-on-flows.rst

@@ -61,7 +61,7 @@ We already have seen the ``start`` and ``await`` keywords to trigger a flow. We
         user said "Hi"
         bot say "Hello again"
 
-Running this example you will see the bot responding with "Hello again" as long as you keep greeting with "Hi":
+By running this example you will see the bot responding with "Hello again" as long as you keep greeting with "Hi":
 
 .. code-block:: text
 
@@ -88,11 +88,30 @@ Running this example you will see the bot responding with "Hello again" as long
 
 In contrast, you can only say "Bye" once before you restart the story.
 
-Activating a flow enables you to keep matching the interaction event sequence against the pattern defined in the flow, even if the pattern previously successfully matched the interaction event sequence (finished) or failed. Since the same flow configuration can only be activated once, you can use the flow activation directly wherever you require the flow's functionality. This on demand pattern is better than activating it once in the beginning before you actually know if it is needed.
+Activating a flow enables you to keep matching the interaction event sequence against the pattern defined in the flow, even if the pattern previously successfully matched the interaction event sequence (finished) or failed. Since the same flow configuration can only be activated once, you can use the flow activation directly wherever you require the flow's functionality. This `on demand pattern` is better than activating it once in the beginning before you actually know if it is needed.
 
 .. important::
     Activating a flow will start a flow and automatically restart it when it has ended (finished or failed) to match to reoccurring interaction patterns.
 
+Alternatively, you can use the ``@active`` decorator notation to activate a flow at the start as a child of the main flow:
+
+.. code-block:: colang
+
+    import core
+
+    flow main
+        bot say "Welcome"
+        user said "Bye"
+        bot say "Goodbye"
+        match RestartEvent()
+
+    @active
+    flow managing user greeting
+        user said "Hi"
+        bot say "Hello again"
+
+If you use this for flows that were defined in a separate Colang library module, they will get automatically activated when the library is imported. But we advice you to use the ``activate`` statement if possible, since it is more explicit and result in better readability.
+
 .. important::
     The main flow behaves also like an activated flow. As soon as it reaches the end it will restart automatically.
 

docs/colang_2/language_reference/python-actions.rst

@@ -28,7 +28,7 @@ And here is how you can call it from a Colang flow:
         $result = await CustomTestAction(value=5)
         bot say "The result is: {$result}"
 
-Alternatively, if you need an asynchronous function you can define it like that:
+Be aware that awaiting in contrast to action in the context of UMIM events Python Actions are blocking per default. That means if the action is implementing a long running task (e.g. an REST API request or) you will want to make the Python Action asynchronous. You can define it by adding the parameter ``execute_async=True`` to the function decorator :
 
 .. code-block:: python
 
@@ -44,16 +44,21 @@ And here is how you can call it from a Colang flow:
 .. code-block:: colang
 
     flow main
+        # Option 1 start the Action and let your flow continue while until you really need the result from the action
         start CustomTestAction(value=5) as $action_ref
         # Some other statements ...
-        await $action_ref.Finished() as $event_ref
+        match $action_ref.Finished() as $event_ref
         bot say "The result is: {$event_ref.return_value}" # Access the function return value via the event reference
 
+        # Option 2: You can still use async Python actions like you would use any other action (the same as for non async Python actions)
+        $result = await CustomTestAction(value=5)
+        bot say "The result is: {$result}"
+
 .. note::
 
     All Python action names need to end with ``Action``.
 
-In addition to all the custom user defined parameters, the following parameters are available in a Python action:
+In addition to all the custom user defined parameters, the parameters below are available in a Python action. To make use of these parameters in your Python action implementation add the parameter to your function signature.
 
 .. code-block:: python
 

docs/colang_2/language_reference/working-with-variables-and-expressions.rst

@@ -86,8 +86,8 @@ Colang supports evaluation of common Python expressions for simple and compound
     ** # to the power of: 2 ** 10 -> 1024
     % # modulus
     ==, <, >, <=, >= # comparison operators
+    and, or, not # logical operators
     in # is something contained within something else
-    not in # is something not contained within something else
     >>, <<, ^, |, &, ~ # Bitwise operators
 
     # Conditional expressions
@@ -117,6 +117,8 @@ Colang supports evaluation of common Python expressions for simple and compound
     is_float(x: Any) -> bool # Check if x is a float
     is_str(x: Any) -> bool # Check if x is a str
     is_regex(x: Any) -> bool # Check if x is a regex pattern
+    type(x: Any) -> str # Returns type as string of object x
+    list(x: Iterable[T]) -> list[T] # Converts an iterable object to a list
     rand() -> float # Return a random float between 0 and 1
     randint(x: int) -> int # Return a random int below x
     flows_info() -> dict # Returns a dictionary that contains more information about the current flow
@@ -133,6 +135,7 @@ Here is how expression can be used withing Colang:
 
     # Expression as a flow parameter
     bot count to ($dict["value"])
+    bot count to (int("3"))
 
 You see how expressions can be used in different context and need to be wrapped in parentheses if used as a *standalone statement* or as a *flow parameter*.
 
@@ -192,32 +195,81 @@ As in Python's formatted string literals we can use braces to evaluate an expres
 If you need to include a brace character in the literal text, it can be escaped by doubling: ``{{`` and ``}}``.
 
 ----------------------------------------
-Built-in Flow Variables
+Flow Member Variables
 ----------------------------------------
 
-.. important::
-    This is work in progress and some of the built-in variables might change or be removed in the future.
+To access a flow instance's member variables you can use a reference or the reserved variable ``$self`` from within the flow itself:
+
+.. code-block:: colang
+
+    $ref.uid: str # The unique id of the flow instance
+    $ref.flow_id: str # The name of the flow
+    $ref.status.value: str # Name of the low state ("waiting", "starting", "started", "stopping", "stopped", "finished")
+    $ref.loop_id: Optional[str] # The interaction loop id of the flow
+    $ref.parent_uid: Optional[str] # The unique id of the parent flow instance
+    $ref.child_flow_uids: List[str] # All unique ids of the child flow instances
+    $ref.context: dict # The variable context that contains all user defined variables in the flow
+    $ref.priority: float # Priority of the flow
+    $ref.arguments: dict # All arguments of the flow
+    $ref.flow_instance_uid: str # Flow instance specific uid
+    $ref.source_flow_instance_uid: str # The parent flow uid of the flow
+    $ref.activate: bool # True if the flow was activated and will therefore restart immediately when finished
+    $ref.new_instance_started: bool # True if new instance was started of an activated flow
+
+You should not change those values if you are not sure what you are doing since this can have side effects on the further execution of the flow!
+
+
+----------------------------------------
+Action Member Variables
+----------------------------------------
+
+To access the members of an action you can use an action reference:
+
+.. code-block:: colang
+
+    $ref.uid: str # The unique id of the action instance
+    $ref.name: str # The name of the action
+    $ref.flow_uid: str # The flow that the action started
+    $ref.status.value: str # The action status ("initialized", "starting", "started", "stopping", "finished")
+    $ref.context: dict # Contains all the action event parameters
+    $ref.start_event_arguments: dict # Contains all action start arguments
+
+----------------------------------------
+Event Member Variables
+----------------------------------------
 
-Currently, there are a couple of variable names that cannot be used as custom variable names in a flow. They contain flow instance specific information:
+To access the members of an event you can use an event reference:
 
 .. code-block:: colang
 
-    $system: dict # System specific data like e.g. the current bot configuration `$system.config`
-    $uid: str # The unique id of the flow instance
-    $flow_id: str # The name of the current flow
-    $loop_id: Optional[str] # The interaction loop id of the current flow
-    $parent_uid: Optional[str] # The unique id of the parent flow instance
-    $child_flow_uids: List[str] # All unique ids of the child flow instances
-    $context: dict # The current variable context that contains all user defined variables in the flow
-    $priority: float # Current priority of the flow
-    $arguments: dict # All arguments of the flow
-    $flow_instance_uid: str # Flow instance specific uid
-    $source_flow_instance_uid: str # The parent flow uid of the flow
-    $activate: bool # True if the flow was activated and will therefore restart immediately when finished
-    $new_instance_started: bool # True if new instance was started of an activated flow
-
-    # Other internal flow members that cannot be used:
-    $hierarchy_position, $heads, $scopes, $head_fork_uids, $action_uids, $global_variables,
-    $status_updated, $source_head_uid
+    $ref.name: str # The name of the event
+    $ref.arguments: str # The name of the event arguments
+
+    # Only for flow events
+    $ref.flow: FlowReference # A reference to the flow of the event
+
+----------------------------------------
+System and Bot Configuration Values
+----------------------------------------
+
+To access system or bot specific configuration variables you can use ``$system``:
+
+.. code-block:: colang
+
+    $system.config # Current bot configuration object (YAML)
+    $system.state # The bots current runtime state object
+
+As an example, if you defined a new boolean value `streaming` in the yaml bot configuration:
+
+.. code-block:: yaml
+
+    streaming: True
+
+you can access and print it like that:
+
+.. code-block:: colang
+
+    print $system.config.streaming
+
 
 Next we learn how to use :ref:`flow-control` to create branching or looping interaction patterns.

docs/colang_2/overview.rst

@@ -14,7 +14,7 @@ Colang is an *event-driven interaction modeling language* that is interpreted by
      - 1.0
    * - 0.8
      - 2.0-alpha
-   * - 0.9
+   * - >= 0.9
      - 2.0-beta
 
 Motivation
@@ -81,6 +81,16 @@ Current limitations (to be fixed in NeMo Guardrails v0.10.0):
 - Guardrails Library is not yet usable from within Colang 2.0.
 - Generation options not supported, e.g. log activated rails, etc.
 
+Migration from alpha to beta version
+------------------------------------
+
+You can migrate your Colang 2.0-alpha bots to 2.0-beta using the following command:
+
+.. code-block:: console
+
+    nemoguardrails convert "path/to/2.0-alpha/version/bots" --from-version "2.0-alpha"
+
+See section :ref:`Breaking changes from alpha to beta version <whats-changed-alpha-to-beta>` to see the detailed changes.
 
 Interaction Model
 =================