feat: add support for Claude 3 tool use (function calling) #1587

dirname · 2024-07-01T09:36:44Z

close #1572, close #1258, close #1504, close #1299, close #1374, close #1520 (claude)

我已确认该 PR 已自测通过，相关截图如下：

分叉情况

在流式消息下存在一些与 OpenAI 分叉的情况

为了便于理解这些差异，以下引用了来自 Claude 官方文档中的流式工具使用响应示例：https://docs.anthropic.com/en/api/messages-streaming#streaming-request-with-tool-use

event: message_start
data: {"type":"message_start","message":{"id":"msg_014p7gG3wDgGV9EUtLvnow3U","type":"message","role":"assistant","model":"claude-3-haiku-20240307","stop_sequence":null,"usage":{"input_tokens":472,"output_tokens":2},"content":[],"stop_reason":null}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: ping
data: {"type": "ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Okay"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" let"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"'s"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" check"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" the"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" weather"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" for"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" San"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" Francisco"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":","}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" CA"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":":"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01T1x1fJ34qAmk2tNTrN7Up6","name":"get_weather","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"location\":"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" \"San"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" Francisc"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"o,"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" CA\""}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":", "}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"\"unit\": \"fah"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"renheit\"}"}}

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":89}}

event: message_stop
data: {"type":"message_stop"}

Arguments 为空

Claude Request Body:

{
    "model": "claude-3-5-sonnet-20240620",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "现在几点了"
                }
            ]
        }
    ],
    "max_tokens": 4096,
    "stream": true,
    "tools": [
        {
            "name": "getCurrentTime",
            "description": "获取当前时间",
            "input_schema": {
                "type": "object",
                "properties": {}
            }
        }
    ],
    "tool_choice": {
        "type": "auto"
    }
}

Claude Response:

OpenAI Request Body:

{
    "model": "gpt-4o",
    "stream": true,
    "messages": [
        {
            "content": "现在几点了",
            "role": "user"
        }
    ],
    "tools": [
        {
            "function": {
                "description": "获取当前时间",
                "name": "getCurrentTime",
                "parameters": {
                    "type": "object",
                    "properties": {}
                }
            },
            "type": "function"
        }
    ]
}

OpenAI Response:

在 OpenAI 流式函数调用 没有 arguments 的情况下，会响应一个 {}，在 Claude 流式中则 什么都不响应 (经过请求测试 content_block_delta 第一个 partial_json 始终为 ""，content_block_start tool_use 的 input 始终为 {} )

commit d16cd615 针对这一情况，当 finish_reason 时，且 arguments 为空的情况下，添加一个 {} 以便更好地与 OpenAI 的行为保持一致，兼容一些应用 (如 LobeChat，若不添加 {} 则无法调用时钟时间)

多余的响应

Claude 在流式响应中包含了 ping, content_block_stop, message_stop 等事件，这可能导致流式传输中出现额外输出。尽管在 LobeChat 的测试中，这些多余的输出并未对函数调用的使用造成影响，但仍可以 由社区讨论确定是否有必要通过过滤逻辑来去除这些额外响应，尤其是当它们开始干扰某些应用功能使用时

已知的问题

在进行请求转换 (ConvertRequest) 时，Claude 的 role 枚举仅能包含 user, assistant refer: https://docs.anthropic.com/en/api/messages

然而 OpenAI 使用 role tool 传递函数调用的结果给模型 refer: https://platform.openai.com/docs/guides/function-calling

Claude 通过 user 来传递 tool_result refer: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#handling-tool-use-and-tool-result-content-blocks

同时 Claude 中 role 需要交替组合，以 LobeChat 的时钟时间调用为例

Request Body:

{
    "model": "claude-3-5-sonnet-20240620",
    "stream": true,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "temperature": 0.6,
    "top_p": 1,
    "messages": [
        {
            "content": "## Tools\n\nYou can use these tools below:\n\n### Clock Time\n\nDisplay a clock to show current time\n\nThe APIs you can use:\n\n#### clock-time____getCurrentTime____standalone\n\n获取当前时间\n\n### Realtime Weather\n\nGet realtime weather information\n\nThe APIs you can use:\n\n#### realtime-weather____fetchCurrentWeather\n\n获取当前天气情况",
            "role": "system"
        },
        {
            "content": "现在几点了",
            "role": "user"
        },
        {
            "content": "为了回答您的问题,我需要使用时钟工具来获取当前的准确时间。让我为您查询一下。",
            "role": "assistant",
            "tool_calls": [
                {
                    "function": {
                        "arguments": "{}",
                        "name": "clock-time____getCurrentTime____standalone"
                    },
                    "id": "toolu_01MTZCkPPSgR9927FddGHgkz",
                    "type": "function"
                }
            ]
        },
        {
            "content": "...",
            "name": "clock-time____getCurrentTime____standalone",
            "role": "tool",
            "tool_call_id": "toolu_01MTZCkPPSgR9927FddGHgkz"
        },
        {
            "content": "你好",
            "role": "user"
        }
    ],
    "tools": [
        {
            "function": {
                "description": "获取当前时间",
                "name": "clock-time____getCurrentTime____standalone",
                "parameters": {
                    "type": "object",
                    "properties": {}
                }
            },
            "type": "function"
        }
    ]
}

在 convert 后，tool 会被替换为 user

{
    ....
        {
            "role": "user",
            "content": [
                {
                    "type": "tool_result",
                    "content": "...",
                    "tool_use_id": "toolu_01MTZCkPPSgR9927FddGHgkz"
                }
            ]
        },
    ....
}

此时出现了连续多个 role 为 user 的情况，报错

{"type":"error","error":{"type":"invalid_request_error","message":"messages: roles must alternate between \"user\" and \"assistant\", but found multiple \"user\" roles in a row"}}

由于客户端侧的情况复杂多样，如果在服务端中重新 convert 请求，可能会引发更多的问题

考虑到函数调用主要是为了向模型提供外部结果，增强其能力和扩展应用场景，对函数调用结果不包含 AI-generated content 属于边缘情形，只会影响特定场景下的使用

该已知问题与 #1135 #1187 相关

ye4293 · 2024-07-03T07:18:29Z

你太强了，我的超人

ai-poet · 2024-07-09T06:07:59Z

单走一个6

…eng#1587) * feat: add tool support for AWS & Claude * fix: add {} for openai compatibility in streaming tool_use

dirname added 2 commits July 1, 2024 16:57

feat: add tool support for AWS & Claude

ea1c293

fix: add {} for openai compatibility in streaming tool_use

d16cd61

songquanpeng merged commit 0fc07ea into songquanpeng:main Jul 1, 2024
3 checks passed

dirname deleted the claude-tools branch July 1, 2024 16:30

mxdlzg pushed a commit to mxdlzg/one-api that referenced this pull request Oct 15, 2024

feat: add support for Claude 3 tool use (function calling) (songquanp…

750b4cc

…eng#1587) * feat: add tool support for AWS & Claude * fix: add {} for openai compatibility in streaming tool_use

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for Claude 3 tool use (function calling) #1587

feat: add support for Claude 3 tool use (function calling) #1587

dirname commented Jul 1, 2024 •

edited by songquanpeng

Loading

ye4293 commented Jul 3, 2024

ai-poet commented Jul 9, 2024

feat: add support for Claude 3 tool use (function calling) #1587

feat: add support for Claude 3 tool use (function calling) #1587

Conversation

dirname commented Jul 1, 2024 • edited by songquanpeng Loading

分叉情况

Arguments 为空

多余的响应

已知的问题

ye4293 commented Jul 3, 2024

ai-poet commented Jul 9, 2024

dirname commented Jul 1, 2024 •

edited by songquanpeng

Loading