Add agent framework/throttling/hidden model/OS assistant and update c…

…onversational search documentation (#6354) * Add agent framework documentation Signed-off-by: Fanit Kolchina <[email protected]> * Add hidden model and API updates Signed-off-by: Fanit Kolchina <[email protected]> * Vale error Signed-off-by: Fanit Kolchina <[email protected]> * Updated field names Signed-off-by: Fanit Kolchina <[email protected]> * Add updating credentials Signed-off-by: Fanit Kolchina <[email protected]> * Added tools table Signed-off-by: Fanit Kolchina <[email protected]> * Add OpenSearch forum thread for OS Assistant Signed-off-by: Fanit Kolchina <[email protected]> * Add tech review for conv search Signed-off-by: Fanit Kolchina <[email protected]> * Fix links Signed-off-by: Fanit Kolchina <[email protected]> * Add tools Signed-off-by: Fanit Kolchina <[email protected]> * Add links to tools Signed-off-by: Fanit Kolchina <[email protected]> * More info about tools Signed-off-by: Fanit Kolchina <[email protected]> * Tool parameters Signed-off-by: Fanit Kolchina <[email protected]> * Update cat-index-tool.md Signed-off-by: kolchfa-aws <[email protected]> * Parameter clarification Signed-off-by: Fanit Kolchina <[email protected]> * Tech review feedback Signed-off-by: Fanit Kolchina <[email protected]> * Typo fix Signed-off-by: Fanit Kolchina <[email protected]> * More tech review feedback: RAG tool Signed-off-by: Fanit Kolchina <[email protected]> * Tech review feedback: memory APis Signed-off-by: Fanit Kolchina <[email protected]> * Update _ml-commons-plugin/agents-tools/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Update _ml-commons-plugin/opensearch-assistant.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Update _ml-commons-plugin/agents-tools/tools/ppl-tool.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Separated search and get APIs and add conversational flow agent Signed-off-by: Fanit Kolchina <[email protected]> * More parameters for PPL tool Signed-off-by: Fanit Kolchina <[email protected]> * Added more parameters Signed-off-by: Fanit Kolchina <[email protected]> * Tech review feedback: PPL tool Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Rename to automating configurations Signed-off-by: Fanit Kolchina <[email protected]> * Editorial comments on the new text Signed-off-by: Fanit Kolchina <[email protected]> * Add parameter to PPl tool Signed-off-by: Fanit Kolchina <[email protected]> * Changed link to configurations Signed-off-by: Fanit Kolchina <[email protected]> * Rate limiter feedback and added warning Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
opensearch-project · Feb 20, 2024 · 3f7468b · 3f7468b
1 parent d706d2e
commit 3f7468b
Show file tree

Hide file tree

Showing 99 changed files with 5,809 additions and 844 deletions.
diff --git a/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt b/.github/vale/styles/Vocab/OpenSearch/Products/accept.txt
@@ -59,6 +59,8 @@ Open Distro
 OpenAI
 OpenID Connect
 OpenSearch
+OpenSearch Assistant
+OpenSearch Assistant Toolkit
 OpenSearch Benchmark
 OpenSearch Dashboards
 OpenSearch Playground

diff --git a/_automating-workflows/api/create-workflow.md → ...ing-configurations/api/create-workflow.md b/_automating-workflows/api/create-workflow.md → ...ing-configurations/api/create-workflow.md
@@ -17,7 +17,7 @@ Creating a workflow adds the content of a workflow template to the flow framewor
 * Workflow step fields with invalid values.
 * Workflow graph (node/edge) configurations containing cycles or with duplicate IDs.
 
-To obtain the validation template for workflow steps, call the [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/).
+To obtain the validation template for workflow steps, call the [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/).
 
 Once a workflow is created, provide its `workflow_id` to other APIs.
 
@@ -50,7 +50,7 @@ POST /_plugins/_flow_framework/workflow?provision=true
 ```
 {% include copy-curl.html %}
 
-When set to `true`, the [Provision Workflow API]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/) is executed immediately following creation. 
+When set to `true`, the [Provision Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/) is executed immediately following creation. 
 
 By default, workflows are validated when they are created to ensure that the syntax is valid and that the graph does not contain cycles. This behavior can be controlled with the `validation` query parameter. If `validation` is set to `all`, OpenSearch performs a complete template validation. Any other value of the `validation` parameter suppresses validation, allowing an incomplete/work-in-progress template to be saved. To disable template validation, set `validation` to `none`:
 

diff --git a/_automating-workflows/api/delete-workflow.md → ...ing-configurations/api/delete-workflow.md b/_automating-workflows/api/delete-workflow.md → ...ing-configurations/api/delete-workflow.md
diff --git a/...ing-workflows/api/deprovision-workflow.md → ...onfigurations/api/deprovision-workflow.md b/...ing-workflows/api/deprovision-workflow.md → ...onfigurations/api/deprovision-workflow.md
@@ -10,7 +10,7 @@ nav_order: 70
 This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/flow-framework/issues/475).    
 {: .warning}
 
-When you no longer need a workflow, you can deprovision its resources. Most workflow steps that create a resource have corresponding workflow steps to reverse that action. To retrieve all resources currently created for a workflow, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/). When you call the Deprovision Workflow API, resources included in the `resources_created` field of the Get Workflow Status API response will be removed using a workflow step corresponding to the one that provisioned them.
+When you no longer need a workflow, you can deprovision its resources. Most workflow steps that create a resource have corresponding workflow steps to reverse that action. To retrieve all resources currently created for a workflow, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). When you call the Deprovision Workflow API, resources included in the `resources_created` field of the Get Workflow Status API response will be removed using a workflow step corresponding to the one that provisioned them.
 
 The workflow executes the provisioning workflow steps in reverse order. If failures occur because of resource dependencies, such as preventing deletion of a registered model if it is still deployed, the workflow attempts retries.
 
@@ -56,6 +56,6 @@ If deprovisioning did not completely remove all resources, OpenSearch responds w
 In some cases, the failure happens because of another dependent resource that took some time to be removed. In this case, you can attempt to send the same request again.
 {: .tip}
 
-To obtain a more detailed deprovisioning status than is provided by the summary in the error response, query the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/). 
+To obtain a more detailed deprovisioning status than is provided by the summary in the error response, query the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). 
 
 On success, the workflow returns to a `NOT_STARTED` state. If some resources have not yet been removed, they are provided in the response.
diff --git a/...ting-workflows/api/get-workflow-status.md → ...configurations/api/get-workflow-status.md b/...ting-workflows/api/get-workflow-status.md → ...configurations/api/get-workflow-status.md
@@ -10,7 +10,7 @@ nav_order: 40
 This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/flow-framework/issues/475).    
 {: .warning}
 
-[Provisioning a workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/) may take a significant amount of time, particularly when the action is associated with OpenSearch indexing operations. The Get Workflow State API permits monitoring of the provisioning deployment status until it is complete.
+[Provisioning a workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/) may take a significant amount of time, particularly when the action is associated with OpenSearch indexing operations. The Get Workflow State API permits monitoring of the provisioning deployment status until it is complete.
 
 ## Path and HTTP methods
 

diff --git a/...ating-workflows/api/get-workflow-steps.md → ...-configurations/api/get-workflow-steps.md b/...ating-workflows/api/get-workflow-steps.md → ...-configurations/api/get-workflow-steps.md
diff --git a/_automating-workflows/api/get-workflow.md → ...mating-configurations/api/get-workflow.md b/_automating-workflows/api/get-workflow.md → ...mating-configurations/api/get-workflow.md
@@ -47,4 +47,4 @@ To retrieve a template in JSON format, specify `Content-Type: application/json`
 curl -XGET "http://localhost:9200/_plugins/_flow_framework/workflow/8xL8bowB8y25Tqfenm50" -H 'Content-Type: application/json'
 ```
 
-OpenSearch responds with the stored template containing the same content as the body of the [create workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/) request. The order of fields in the returned template may not exactly match the original template but will function identically.
+OpenSearch responds with the stored template containing the same content as the body of the [create workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/) request. The order of fields in the returned template may not exactly match the original template but will function identically.
diff --git a/_automating-workflows/api/index.md → _automating-configurations/api/index.md b/_automating-workflows/api/index.md → _automating-configurations/api/index.md
@@ -13,11 +13,11 @@ This is an experimental feature and is not recommended for use in a production e
 
 OpenSearch supports the following workflow APIs:
 
-* [Create or update workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/)
-* [Get workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow/)
-* [Provision workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/)
-* [Get workflow status]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/)
-* [Get workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/)
-* [Search workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/search-workflow/)
-* [Deprovision workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/deprovision-workflow/)
-* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/delete-workflow/)
+* [Create or update workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/)
+* [Get workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow/)
+* [Provision workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/)
+* [Get workflow status]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/)
+* [Get workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/)
+* [Search workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/search-workflow/)
+* [Deprovision workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/deprovision-workflow/)
+* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/delete-workflow/)
diff --git a/...ating-workflows/api/provision-workflow.md → ...-configurations/api/provision-workflow.md b/...ating-workflows/api/provision-workflow.md → ...-configurations/api/provision-workflow.md
@@ -12,7 +12,7 @@ This is an experimental feature and is not recommended for use in a production e
 
 Provisioning a workflow is a one-time setup process usually performed by a cluster administrator to create resources that will be used by end users.  
 
-The `workflows` template field may contain multiple workflows. The workflow with the `provision` key can be executed with this API. This API is also executed when the [Create or Update Workflow API]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/) is called with the `provision` parameter set to `true`.
+The `workflows` template field may contain multiple workflows. The workflow with the `provision` key can be executed with this API. This API is also executed when the [Create or Update Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/) is called with the `provision` parameter set to `true`.
 
 You can only provision a workflow if it has not yet been provisioned. Deprovision the workflow if you need to repeat provisioning.
 {: .note}
@@ -48,4 +48,4 @@ OpenSearch responds with the same `workflow_id` that was used in the request:
 }
 ```
 
-To obtain the provisioning status, query the [Get Workflow State API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/).
+To obtain the provisioning status, query the [Get Workflow State API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/).
diff --git a/_automating-workflows/api/search-workflow.md → ...ing-configurations/api/search-workflow.md b/_automating-workflows/api/search-workflow.md → ...ing-configurations/api/search-workflow.md
diff --git a/_automating-workflows/index.md → _automating-configurations/index.md b/_automating-workflows/index.md → _automating-configurations/index.md
@@ -1,13 +1,13 @@
 ---
 layout: default
-title: Automating workflows
+title: Automating configurations
 nav_order: 1
 has_children: false
 nav_exclude: true
-redirect_from: /automating-workflows/
+redirect_from: /automating-configurations/
 ---
 
-# Automating workflows
+# Automating configurations
 **Introduced 2.12**
 {: .label .label-purple }
 
@@ -16,7 +16,7 @@ This is an experimental feature and is not recommended for use in a production e
 
 You can automate complex OpenSearch setup and preprocessing tasks by providing templates for common use cases. For example, automating machine learning (ML) setup tasks streamlines the use of OpenSearch ML offerings.
 
-In OpenSearch 2.12, workflow automation is limited to ML tasks.
+In OpenSearch 2.12, configuration automation is limited to ML tasks.
 {: .info}
 
 OpenSearch use case templates provide a compact description of the setup process in a JSON or YAML document. These templates describe automated workflow configurations for conversational chat or query generation, AI connectors, tools, agents, and other components that prepare OpenSearch as a backend for generative models. For template examples, see [Sample templates](https://github.com/opensearch-project/flow-framework/tree/main/sample-templates).
@@ -38,12 +38,12 @@ Workflow automation provides the following benefits:
 * **Workflows**: One or more workflows containing the following elements:
     * **User input**: Parameters expected from the user that are specific to the steps in this workflow.
     * **Workflow Steps**: The workflow steps described as a directed acyclic graph (DAG):  
-        * ***Nodes*** describe steps of the process, which may be executed in parallel. For the syntax of workflow steps, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-steps/). 
+        * ***Nodes*** describe steps of the process, which may be executed in parallel. For the syntax of workflow steps, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-steps/). 
         * ***Edges*** sequence nodes to be executed after the previous step is complete and may use the output fields of previous steps. When a node includes a key in the `previous_node_input` map referring to a previous node’s workflow step, a corresponding edge is automatically added to the template during parsing and may be omitted for the sake of simplicity.
 
 ## Next steps
 
-- For supported APIs, see [Workflow APIs]({{site.url}}{{site.baseurl}}/automating-workflows/api/index/).
-- For the workflow step syntax, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-steps/).  
-- For a complete example, see [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-tutorial/).
-- For configurable settings, see [Workflow settings]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-settings/).
+- For supported APIs, see [Workflow APIs]({{site.url}}{{site.baseurl}}/automating-configurations/api/index/).
+- For the workflow step syntax, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-steps/).  
+- For a complete example, see [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/).
+- For configurable settings, see [Workflow settings]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-settings/).
diff --git a/_automating-workflows/workflow-settings.md → ...ating-configurations/workflow-settings.md b/_automating-workflows/workflow-settings.md → ...ating-configurations/workflow-settings.md
diff --git a/_automating-workflows/workflow-steps.md → _automating-configurations/workflow-steps.md b/_automating-workflows/workflow-steps.md → _automating-configurations/workflow-steps.md
@@ -20,7 +20,7 @@ Workflow steps are actively being developed to expand automation capabilities. W
 |`id`	|String	|Required	| A user-provided ID for the step. The ID must be unique within a given workflow and is useful for identifying resources created by the step. For example, a `register_agent` step may return an `agent_id` that has been registered. Using this ID, you can determine which step produced which resource.	|
 |`type`	|String	|Required	|The type of action to take, such as `deploy_model`, which corresponds to the API for which the step is used. Multiple steps may share the same type but must each have their own unique ID. For a list of supported types, see [Workflow step types](#workflow-step-types).	|
 |`previous_node_inputs`	|Object	|Optional	| A key-value map specifying user inputs that are produced by a previous step in the workflow. For each key-value pair, the key is the previous step's `id` and the value is an API body field name (such as `model_id`) that will be produced as an output of a previous step in the workflow. For example, `register_remote_model` (key) may produce a `model_id` (value) that is required for a subsequent `deploy_model` step. <br> A graph edge is automatically added to the workflow connecting the previous step's key as the source and the current node as the destination. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
-|`user_inputs`	|Object	|Optional	| A key-value map of inputs supported by the corresponding API for this specific step. Some inputs are required for an API, while others are optional. Required inputs may be specified here, if known, or in the `previous_node_inputs` field. The [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/) identifies required inputs and step outputs. <br> Substitutions are supported in string values, lists of strings, and maps with string values. The pattern `{% raw %}${{previous_step_id.output_key}}{% endraw %}` will be replaced by the value in the previous step's output with the given key.  For example, if a parameter map in the user inputs includes a key `embedding_model_id` with a value `{% raw %}${{deploy_embedding_model.model_id}}{% endraw %}`, then the `model_id` output of the `deploy_embedding_model` step will be substituted here. This performs a similar function to the `previous_node_input` map but is not validated and does not automatically infer edges. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
+|`user_inputs`	|Object	|Optional	| A key-value map of inputs supported by the corresponding API for this specific step. Some inputs are required for an API, while others are optional. Required inputs may be specified here, if known, or in the `previous_node_inputs` field. The [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/) identifies required inputs and step outputs. <br> Substitutions are supported in string values, lists of strings, and maps with string values. The pattern `{% raw %}${{previous_step_id.output_key}}{% endraw %}` will be replaced by the value in the previous step's output with the given key.  For example, if a parameter map in the user inputs includes a key `embedding_model_id` with a value `{% raw %}${{deploy_embedding_model.model_id}}{% endraw %}`, then the `model_id` output of the `deploy_embedding_model` step will be substituted here. This performs a similar function to the `previous_node_input` map but is not validated and does not automatically infer edges. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
 
 ## Workflow step types
 
@@ -61,4 +61,4 @@ You can include the following additional fields in the `previous_node_inputs` fi
 
 ## Example workflow steps
 
-For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-tutorial/).
+For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/).
diff --git a/_automating-workflows/workflow-tutorial.md → ...ating-configurations/workflow-tutorial.md b/_automating-workflows/workflow-tutorial.md → ...ating-configurations/workflow-tutorial.md
diff --git a/_config.yml b/_config.yml
@@ -112,7 +112,7 @@ collections:
   about:
     permalink: /:collection/:path/
     output: true
-  automating-workflows:
+  automating-configurations:
     permalink: /:collection/:path/
     output: true
   dashboards-assistant:
@@ -172,11 +172,8 @@ opensearch_collection:
     ml-commons-plugin:
       name: Machine learning
       nav_fold: true
-    dashboards-assistant:
-      name: Dashboards assistant
-      nav_fold: true
-    automating-workflows:
-      name: Automating workflows
+    automating-configurations:
+      name: Automating configurations
       nav_fold: true
     monitoring-your-cluster:
       name: Monitoring your cluster