[BUG] KnowledgeBaseTool may return parameter placeholder not filled error

[yuye-aws]

What is the bug?
I am implementing the the knowledge base tool. Unfortunately, I meet with a tricky bug from the connector. I did some deep dive and found the error is originating from validatePayload function from Connector.java.

How can one reproduce the bug?
Prerequisites: configure LLM and bedrock knowledge base, and then obtain their model_id.

1. Register the agent with knowledge base tool.

POST /_plugins/_ml/agents/_register
{
  "name": "Test_Agent_For_ReAct_KB",
  "type": "conversational",
  "description": "this is a test agent for knowledge base",
  "llm": {
    "model_id": "rBZl3pABQHtbJkH1MFP3",
    "parameters": {
      "max_iteration": 10,
      "stop_when_no_tool_found": true,
      "response_filter": "$.completion"
    }
  },
  "memory": {
    "type": "conversation_index"
  },
  "tools": [
    {
      "type": "KnowledgeBaseTool",
      "name": "OpensearchKnowledgeBaseTool",
      "description": "A tool to search the Opensearch knowledge base, the knowledge base consists of segments of OpenSearch documents. You should always search data with this tool when encountering general questions about Opensearch. But for questions about current concerete cluster, use this tool can not help you. If this tool provides useful info, give the answer and also quote the origin doc. If this tool can not provide knowledge you need, give answer based on your own knowledge. Action Input: {\"text\": <natrual language keywords for question>}",
      "parameters": {
        "model_id": "phZk3pABQHtbJkH1fVPy"
      }
    },
    {
      "type": "CatIndexTool",
      "name": "RetrieveIndexMetaTool",
      "description": "Use this tool to get OpenSearch index information: (health, status, index, uuid, primary count, replica count, docs.count, docs.deleted, store.size, primary.store.size)."
    }
  ],
  "app_type": "my app"
}

2. Execute the agent to retrieve documents on "agent framework".

POST /_plugins/_ml/agents/_haq3pABQHtbJkH1eVNE/_execute
{
  "parameters": {
    "question": "Use OpensearchKnowledgeBaseTool to get information about agent framework in opensearch",
    "verbose": true
  }
}

3. Receive the following error:

"{\"error\":{\"reason\":\"Invalid Request\",\"details\":\"Some parameter placeholder not filled in payload: VectorDBTool.output\",\"type\":\"IllegalArgumentException\"},\"status\":400}"

What is the expected behavior?
The retrieved documents from bedrock knowledge should be free from payload validation in connector.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[yuye-aws]

Here is one of the documents retrieved from the knowledge base. The substring ${parameters.VectorDBTool.output} matches the regex "\\$\\{parameters\\.([^}]+)}" in

ml-commons/common/src/main/java/org/opensearch/ml/common/connector/Connector.java

Line 82 in 9b413a7

Pattern pattern = Pattern.compile("\\$\\{parameters\\.([^}]+)}");

\n\n Context:\n${parameters.VectorDBTool.output}\n\nHuman:Use OpensearchKnowledgeBaseTool to get information about agent framework in opensearch\n\nAssistant:"       }     }   ] } ``` {% include copy-curl.html %}  OpenSearch returns an agent ID for the newly created agent:  ```json {   "agent_id": "879v9YwBjWKCe6Kg12Tx" } ```  You can inspect the agent by sending a request to the `agents` endpoint and providing the agent ID:  ```json GET \/_plugins\/_ml\/agents\/879v9YwBjWKCe6Kg12Tx ``` {% include copy-curl.html %}  To execute the agent, send the following request. When registering the agent, you configured it to take in `parameters.question`, so you need to provide this parameter in the request.

[yuye-aws]

This function may wrongly replace parameters from retrieved documents from knowledge base tool:

ml-commons/common/src/main/java/org/opensearch/ml/common/connector/HttpConnector.java

Line 311 in 9b413a7

public <T> T createPayload(String action, Map<String, String> parameters) {

[yuye-aws]

Raised a draft PR to fix it: #2724. More discussion is needed.

[jngz-es]

Correct me, if I misunderstand. You mean the output of KnowledgeBaseTool can contain something like the string "${parameters.VectorDBTool.output}", right?

[yuye-aws]

Correct me, if I misunderstand. You mean the output of KnowledgeBaseTool can contain something like the string "${parameters.VectorDBTool.output}", right?

Yes.

[yuye-aws]

The bug also persists with remote LLM connector:

POST /_plugins/_ml/models/<remote_llm_model_id>/_predict/
{
  "parameters": {
    "prompt": """Human: What is ${parameters.foo} \n Assistant:"""
  }
}

And receives error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Some parameter placeholder not filled in payload: foo"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Some parameter placeholder not filled in payload: foo"
  },
  "status": 400
}

[yuye-aws]

Created an RFC as a solution: #2748