[RFC] 018 - 多模型服务商一期：架构设计 & AWS Bedrock / Zhipu / Gemini / Moonshot 支持 #737
Replies: 13 comments 14 replies
-
技术选型讨论AI SDK VS LangChain 大的方向上，我觉得 chatCompletion 这个模块仍然基于 OpenAI 和 Vercel AI SDK ，不用 LangChain 了，流式输出 functionCall 的问题已经过去快一周了，官方都没解决。 另外，现在整个社区的趋势就是接口调用的风格和规范会尽量对齐 OpenAI 的 chat completion。因此用 openai + vercel ai sdk 基本上可以满足 80% 主流模型的诉求 。
|
Beta Was this translation helpful? Give feedback.
-
对话重构为 LangChain 实现坑点 1：LangChain 官方实现对 stream 中的错误处理不完善参考官方文档，可以使用 try {
const parser = new HttpResponseOutputParser();
const stream = await model.pipe(parser).stream(messages);
return new Response(stream);
} catch (error) {
// error handle
}如果请求是正常的，那么上述代码可以正常运行。 但一个巨坑的地方是如果stream 出错，catch 不会捕获错误，会直接退出运行。 已提 issue : langchain-ai/langchainjs#4015 。 坑点2：
|
Beta Was this translation helpful? Give feedback.
-
错误处理记录Zhipu初始化
Chat 运行态
{
"message": "test",
"name": "TypeError",
"stack": "TypeError: test\n at LobeZhipuAI.chat (webpack-internal:///(rsc)/./src/libs/agent-runtime/zhipu/index.ts:53:19)\n"
} |
Beta Was this translation helpful? Give feedback.
-
|
@arvinxx The error handling with stream should be: stream.on('error', (e) => handleError(e))
.pipe(b)
.on('error', (e) => handleError(e))
.pipe(c)
.on('error', (e) => handleError(e));
function handleError(e) { throw e; } // Maybe also some logging or smth. |
Beta Was this translation helpful? Give feedback.
-
|
thanks, I will try~ |
Beta Was this translation helpful? Give feedback.
-
|
@snowyu it doesn't work. But LangChain have solved this problem officially. |
Beta Was this translation helpful? Give feedback.
-
节奏分为三期进行：
|
Beta Was this translation helpful? Give feedback.
-
LogoGemini https://icon-sets.iconify.design/logos/google-gemini/ |
Beta Was this translation helpful? Give feedback.
-
|
约等于 google bard, gemini 的 logo 直接做 icon感觉太大了 Google Bard: https://simpleicons.org/?q=google+bard ChatGLM 的他们还没传 simpleicons 么 |
Beta Was this translation helpful? Give feedback.
-
|
想搞个 llm logo & icon 仓库了 |
Beta Was this translation helpful? Give feedback.
-
|
要分两种 一种是model tag，我初步搞好了，在 https://github.com/lobehub/lobe-chat/blob/feat/model-provider/src/components/ModelTag/index.tsx 另外一种在设置里配置模型服务商，倒是可以用这种大的，更容易凸显品牌 |
Beta Was this translation helpful? Give feedback.
-
实现思路整体思路
LobeAIProvider支持接口
错误类型
|
Beta Was this translation helpful? Give feedback.
-
|
本次架构升级的考虑要点 模型元信息维护需要有一个全局视角来维护与管理模型信息，包括 model id ，展示名称，maxToken ，是否支持function call 与 vision 等。 有了这个信息以后，我们就可以有一个更好的全局视角来管理模型 错误处理框架和单一的模型服务商不再一样，需要精细化梳理各个模型服务商提供的错误类型，然后找到公共部分与差异化部分，并用一个统一的错误处理方案来承接。 服务端配置、客户端配置的合并与可视化配置目前已经实现了从服务端读取配置作为客户端的默认值，并且支持用户在前端界面进行个性化配置 这里后续需要重点的是服务端可控性，提供一种灵活的方式允许服务端是否开启或关闭 OneAPI 兼容性目前存在大量的用户仍在使用 oneapi 的方案来统一管理模型，我们需要提供一种机制来让这部分用户仍然可以进行相应的配置与管理。 |
Beta Was this translation helpful? Give feedback.
-
|
需要注意的点：
|
Beta Was this translation helpful? Give feedback.
-
|
对于配置表单UI建议用JSONSchema统一生成: https://github.com/rjsf-team/react-jsonschema-form/ |
Beta Was this translation helpful? Give feedback.
-
|
感谢建议，后续可以试试看 |
Beta Was this translation helpful? Give feedback.
-
Zhipu AI 接入踩坑点记录由于智谱 V4 的API 朝 OpenAI 对齐了，所以整体接入比较顺利，不过仍然存在几个坑点： 图片base64需要移除前缀；智谱不支持
|
Beta Was this translation helpful? Give feedback.
-
|
一期结论：针对 GLM 系列模型，本次上线版本先不放出 FC 功能，等智谱侧修复后再上线 Vision Model 由于集成 Google Genimini 时做了一下 base64 的解析与转换，因此可以复用实现兼容，等到时候同样更新修复后移除 |
Beta Was this translation helpful? Give feedback.
-
|
放弃了，等他官方支持再做流式 |
Beta Was this translation helpful? Give feedback.
-
AWS Bedrock 接入笔记基本上参考 https://sdk.vercel.ai/docs/guides/providers/aws-bedrock 来接入会比较顺利。 初始化client: const bedrockClient = new BedrockRuntimeClient({
region: process.env.AWS_REGION ?? 'us-east-1',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID ?? '',
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY ?? '',
},
});chat 接口： // Ask Claude for a streaming chat completion given the prompt
const bedrockResponse = await bedrockClient.send(
new InvokeModelWithResponseStreamCommand({
modelId: 'anthropic.claude-v2',
contentType: 'application/json',
accept: 'application/json',
body: JSON.stringify({
prompt: experimental_buildAnthropicPrompt(messages),
max_tokens_to_sample: 300,
}),
}),
);
// Convert the response into a friendly text-stream
const stream = AWSBedrockAnthropicStream(bedrockResponse);
// Respond with the stream
return new StreamingTextResponse(stream);其中 但在接入 llama2 时则遇到了问题， llama 的 prompt 结构和 aws 的会有很大差别，比如 {
body: {
httpStatusCode: 400,
requestId: '*',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
message: 'Malformed input request: #: extraneous key [max_tokens_to_sample] is not permitted, please reformat your input and try again.',
region: 'us-east-1',
type: 'ValidationException'
}所以计划一期先不上 llama2。 |
Beta Was this translation helpful? Give feedback.
-
|
更新：Vercel AI SDK 有 llama 的示例： https://sdk.vercel.ai/docs/api-reference/aws-bedrock-stream#example-llama-2-models 。试了下可以用，因此本期可以上。 |
Beta Was this translation helpful? Give feedback.
-
Google Gemini 接入核心是代理问题，所以参考 google-gemini/generative-ai-js#29 (comment) 在 provider route 加了一个 proxy，在 proxy 模式下，使用 nodejs runtime，线上仍然使用 edge runtime // due to the Chinese region does not support accessing Google / OpenAI
// we need to use proxy to access it
const proxyUrl = process.env.HTTP_PROXY_URL;
const useProxy = !!proxyUrl;
if (useProxy) {
const { setGlobalDispatcher, ProxyAgent } = require('undici');
setGlobalDispatcher(new ProxyAgent({ uri: proxyUrl }));
}
// undici only can be used in NodeJS.
// So when using proxy, switch to NodeJS runtime
export const runtime = useProxy ? 'nodejs' : 'edge'; |
Beta Was this translation helpful? Give feedback.
-
后续如何集成一个新 ProvidermodelProviders 元信息
import { ModelProviderCard } from '@/types/llm';
const ZhiPu: ModelProviderCard = {
chatModels: [
{
description: '最新的 GLM-4 、最大支持 128k 上下文、支持 Function Call 、Retreival',
displayName: 'GLM-4',
// functionCall: true,
id: 'glm-4',
tokens: 128_000,
},
{
description:
'实现了视觉语言特征的深度融合，支持视觉问答、图像字幕、视觉定位、复杂目标检测等各类多模态理解任务',
displayName: 'GLM-4 Vision',
id: 'glm-4v',
tokens: 128_000,
vision: true,
},
{
description: '最新的glm-3-turbo、最大支持 128k上下文、支持Function Call、Retreival',
displayName: 'GLM-3 Turbo',
// functionCall: true,
id: 'glm-3-turbo',
tokens: 128_000,
},
],
id: 'zhipu',
};
export default ZhiPu;服务端实现添加环境变量在 新增runtime在 runtime 新增实现 LobeRuntimeAI， 需要包包含初始化、 chat 接口，并需要实现初始化错误、 业务逻辑错误这两个错误处理 路由集成在 chat/provider/route 接入实现 默认配置集成在 api/config 设定发给客户端是否开启的属性 客户端实现模型/服务商 iconComponents/ModelIcon 补充判断逻辑 selectors添加是否启用的逻辑 settings增加一个设置面板渲染组件 api配置面板在error/apikeyform新增配置表单 服务端：
客户端：
|
Beta Was this translation helpful? Give feedback.
-
|
Really helpful, but I can't find the flow to update locales p.s: I don't have a valid OPEN_AI_KEY yet |
Beta Was this translation helpful? Give feedback.
-
|
@arvinxx, just want to say "Thank you!" not only for all the changes you have maintained, but also for the knowledge you have shared. The steps to add Moonshot really helped me understand the overall workflow around onboarding a new model provider. Keep up the good work! |
Beta Was this translation helpful? Give feedback.
-
特性跟踪Google Gemini 支持
但个人感觉 Gemini Pro Vision 的视觉能力非常垃圾，还不支持多轮对话，根本没法在日常中使用 AWS Bedrock 支持
智谱 GLM 支持
个人试用下来真的比 Gemini 、Bedrock 之类的好太多了！！！ 国货之光！！！ Azure OpenAI 接入重构TODO |
Beta Was this translation helpful? Give feedback.
-
API Key 填写
对比
|
Beta Was this translation helpful? Give feedback.
-
背景
随着 LobeChat 的发展，社区同学对模型服务商的多样性也产生了新的诉求。我们不能只锚着 OpenAI ，而是要丰富模型服务商的多样性，为用户提供更多的会话选择。
Beta Was this translation helpful? Give feedback.
All reactions