-
Notifications
You must be signed in to change notification settings - Fork 917
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] No-code designer for AI-augmented workflows #4755
Comments
Thanks for sharing this. My main comment is less on implementation and more about why users would like this in OpenSearch and the use cases for it in this platform. I'm having trouble identifying what "common AI/ML use cases" we are trying to address here, given we already have ways to do semantic search and conversational search/question answering that are easy for customers to configure and use. Although projects like LangChain exist to build LLM applications, it's not clear to me that users are looking to run that type of component in an OpenSearch cluster for arbitrary AI apps. I would recommend adding some specific examples and use cases from users to why they want this functionality specifically in OpenSearch (a search platform). Otherwise, if we cannot find these use cases, it may be worth creating this as an experimental plugin and seeing if it gets traction first (e.g. do customers want to run these components in OpenSearch or use LangChain outside of the cluster in their application stack). |
@jonfritz thanks for taking a look!
The perspective here is that these use cases (e.g., semantic search, RAG) are not easy to configure and use. The primary purpose of the backend framework (see RFC) is to automate some of this infrastructure setup, while the primary purpose of a drag-and-drop UI is to help visualize and construct the infrastructure needed.
A big motivation for having this framework, along with in-cluster AI/ML components, is to remove application builders from needing to have middleware infrastructure for running them; the entire workflows can happen within OpenSearch. Feel free to leave comments on that RFC which may be a better place for discussion, since this is more about the drag-and-drop UI of the backend framework components. |
Note: this RFC is for the no-code frontend associated with the proposed AI workflow framework RFC.
Proposal
In the proposed AI workflow framework RFC, we detail how we can simplify the configuration and creation of complex ML use cases by interfacing them in the form of use-case templates. The frontend no-code designer can simplify the creation of these use-case templates via a drag-and-drop interface, similar to existing solutions built on top of LangChain, such as Flowise or LangFlow. This will provide users an intuitive no-code alternative to create and edit common AI-augmented workflows, and to enable rapid prototyping and validation of a created flow.
We aim to keep this designer very lightweight. It can be viewed simply as a means of interacting with use case templates in a visual manner. It is a thin interface built on top of the use-case templates, such that the backend plugin can be fully sufficient for users, and the frontend plugin can be ignored entirely if users exclusively want to interact via APIs.
Additionally, we will provide a set of pre-defined templates for common use cases, such as semantic search or RAG. This will give users a great starting point to be able to quickly develop their applications, and get familiar with the available ML offerings OpenSearch provides.
Goals and benefits
The goal is very similar to those outlined in the proposed AI framework RFC - we want to simplify the complex setup around creating, configuring, and modifying available AI-augmented application workflows provided by OpenSearch. While the backend plugin is focused on providing a framework to automate complex use cases, the frontend plugin is focused on providing an easy-to-use user interface for creating, configuring, editing, and importing such use cases via a drag-and-drop experience.
This will greatly benefit users by providing a visual method of interacting with the templates, and provide further abstraction to the low-level workflow details. It will let users (including those with little to no expertise) be able to leverage the complex and growing ecosystem of AI/ML offerings within OpenSearch via out-of-the-box workflows that users can utilize as a starting point for their own workflows.
Background
Over the last several OpenSearch releases, there have been a growing number of building blocks and tools added to enable AI-augmented workflows, such as semantic search. This is accomplished by configuring and piecing together components from many different plugins, such as the k-NN plugin, ML-commons plugin, neural search plugin, and others planned for the future. For more details, see the “Background” section in the proposed AI workflow framework.
At a high level, the backend framework will simplify the creation of common AI/ML use cases by automating the steps needed to execute such use cases. For example, the current semantic search use case requires configuring a text embedding processor, ingest pipeline, embedding model, k-NN index, and constructing neural queries linking together all of these components to execute the workflow. We can simplify this through a single JSON template that the backend framework can process to construct a workflow, orchestrating the sequential flow of API calls from different plugins. A workflow ID will be returned, which can be used to interface with the workflow by passing the specified input (e.g., plain text for a semantic search workflow). Even though these templates are already an abstraction, they can still become quite complex depending on the use case. This is where a UI can provide a simple way for users to configure and create, import, and edit such templates, and help bridge that gap.
High level design
As stated previously, we aim to simplify the design as much as possible, and treat this plugin as a lightweight way of viewing, configuring, and creating use-cases in this templated format that the backend plugin can parse and execute. We can do this through a drag-and-drop style interface allowing users to pick and choose different building blocks and construct an end-to-end workflow. We can take inspiration from popular existing solutions for building large language model (LLM)-based applications, such as Flowise or LangFlow. These applications have 5 major components:
We can follow a similar design for our plugin, where the individual nodes/components are a set of available resources within OpenSearch, such as deployed ML models/connectors, search pipelines, ingest pipelines, processors, indices, etc. These components can have different levels of abstraction, such that users with little to no experience may still be able to wire together a complex workflow, while also allowing advanced users to drill down and see the underlying interconnected subcomponents. This is explained in more detail below.
Proposed implementation
We can break down the implementation into two main portions:
1. Drag-and-drop workspace
There are two main options for this portion:
We prefer the ReactFlow option for several reasons:
2. Components & workflows
We want to provide an experience as simple and abstracted as possible, while still allowing users to drill down and customize individual details within their application. We can accomplish this using concepts of components and workflows:
Component: an OpenSearch-specific resource such as an ML model, a search pipeline, an index, or an ML-commons tool or agent, that will have specified inputs and outputs. Components can be nested inside of other components.
Workflow: an end-to-end application consisting of a set of components stitched together representing a single use case, such as semantic search
We can use the semantic search use case as an example. Suppose a user wants to leverage OpenSearch’s neural search plugin to create a plaintext search application on their website using OpenSearch as the vector store. For a user to configure this manually, it requires many individual steps, and low-level resource creation using several different APIs (for more details, see the documentation). A breakdown with each resource’s dependencies is shown below:
Using this new plugin, users can quickly configure and create the same low-level resources using components: At the highest level, we can have a “Neural search” component, that has a single input and output, both of which are plaintext. It has a set of required and optional fields the user can fill out, and allow the framework to handle all of the underlying creation and configuration:
We can allow users a mechanism to drill down into this component, and see lower-level subcomponents. This may look something like this:
Breaking down further: we can see each individual component:
Component types
We can persist a set of different component types, each having their own set of required & optional fields, input types, output types, styling, and in-component-creation logic (see “In-component creation” below for details). These will be a 1:1 mapping of categories shown in the component catalog where users can drag and drop components into the workspace. Using the example above, we can show what a Neural Query component may look like:
In-component creation
Eventually, we will want to expand functionality such that all resources for a particular component can be created entirely within the component itself. For example, having a “create new” option on an embedding model component would mean supporting the creation of underlying low-level OpenSearch resources (AI connectors, third party service infrastructure configuration, model group permissions, etc.) entirely within the component. This removes the total number of steps and configuration the user needs to perform in order to use the plugin.
Some of these complex creation logic flows are not planned to be supported initially. Some components, such as individual ingest pipelines or k-NN indices, may be supported implicitly by filling out creation-related input fields within the component, and executing the workflow. The scope of how and what components should initially support this is an open question.
Serialization / deserialization
An important aspect of this plugin is the deconstruction and reconstruction of these flows into a readable JSON format to pass via API to the backend plugin. By default, we can get a ReactFlow JSON object by calling toObject() on a ReactFlow Instance:
NodeData and EdgeData have many optional parameters, but commonly includes the following. Note we include some of the subflow-related fields which allow us to support nested nodes. We also show the Viewport which indicates the current position of the workspace:
Suppose we have a workflow that connects an embedding model to a neural index to use for semantic search. We could output something roughly like the following:
ReactFlow provides common examples of saving/restoring using the toObject() fn in callback helper fns. We can do something similar, along with any other metadata needed, and format into what is expected for the backend API.
Note that for all of the custom nodes and styling, we can persist exclusively on the frontend. In the documentation, you can see that nodeTypes is passed as a standalone field when constructing the ReactFlow instance. So, when serializing or deserializing, we can strip away or add nodeTypes, respectively. We can persist each node’s type via type field within each node.
Additional logic will be needed for converting a ReactFlow JSON representation into the interfaced use-case template that will be understood by the backend framework. The exact format of that template and what it will contain is still being discussed.
Open questions
The text was updated successfully, but these errors were encountered: