Conversation search support for Message APIs

[bouhajer]

Limitation on Conversational search support for Messages API with Claude 3.x family

I am building a public workshop for a public event and the purpose of the workshop is to build OpenSearch conversational search with Bedrock Claude 3.5 Sonnet, but this is not possible with the current Conversational search LLM connector.

The current Conversational search in OpenSearch supports Text Completions API for Anthropic Claude, however, Claude 3.x family only works with Messages API.

Anthropic Claude models in Amazon Bedrock from Claude v1 up to Claude v3.5, all support Messages API so we should update the plugin to support Messages API instead of Text Completions API (supported only by Claude v1 and v2).

Customers are interested in building RAG within OpenSearch to create conversational applications and we should make sure that the conversational search supports the state of the art APIs with foundational models.

Tutorial to be updated:

• https://github.com/opensearch-project/ml-commons/blob/main/docs/tutorials/conversational_search/conversational_search_with_Cohere_Command.md

• https://github.com/opensearch-project/ml-commons/blob/main/docs/tutorials/agent_framework/RAG_with_conversational_flow_agent.md

Anthropic Claude Messages API: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html

Thanks

[Zhangxunmt]

This is a workshop required in the 2024 re:invent too.

[austintlee]

https://docs.anthropic.com/en/api/migrating-from-text-completions-to-messages

So, basically, they are moving to an array of messages like OpenAI did when they updated their chat completion API.

[mingshl]

@b4sjoo if this is going to be connected through bedrock, we need to think about adding the model interface as well.

[austintlee]

I'm going to add support for Bedrock's "converse" endpoint. This will allow connector to talk to Claude 3.5 sonnet via Bedrock.

Do we need to support direct connections to Anthropic? I guess there will be users who use Anthropic, but not AWS?

[ylwu-amzn]

Do we need to support direct connections to Anthropic? I guess there will be users who use Anthropic, but not AWS?

I think so, we should also support customers who use Anthropic directly

[austintlee]

I'm just documenting some of the issues and limitations of Bedrock Converse API.

When I use the document with Claude 3.5 sonnet, I get this error:

This model doesn't support documents.

This is not clear from Bedrock's documentation. But for both OpenAI 4o and Anthropic 3.5-sonnet, I only see examples for text and image.

I am, however, able to use document if I use Claude 3 sonnet.