Unable to use non-Gemini models through ChatVertexAI #5196
Comments
|
To address the issue of using Anthropic Claude 3 with LangChain.js, particularly with the Here's a concise guide to get you started:
This approach allows you to leverage Claude 3's capabilities within LangChain.js applications. For detailed implementation and examples, refer to the Keep in mind, this solution circumvents the
|
dosubot
bot
added
the
auto:bug
|
Similar to #2562 The issue is that Claude in Vertex AI is neither the same protocol as Gemini, nor quite the same as what is offered through Anthropic. So we need something that combines the two. Do-able, but hasn't risen in priority. My general thoughts about how to do this would be
The good news is that the authentication part is handled by default with the Connection classes. |
|
+1, would be a very useful capability. Seems like it's already supported in BedrockChat, which is great. |
|
+1, especially given the renewed interest in Claude 3.5 Sonnet |
|
+1 for Claude 3.5 Sonnet |
|
+1 another vote - Claude 3.5 support through Vertex AI is a key integration for companies on GCP to adopt Langchain. |
|
I see the +1s - this is now next on my priority list! I'm aiming to get #5835 into a stable state ASAP so it can be merged, since it contains some updates that would be useful for this effort as well. The approach I outlined above still mostly holds. Sorry I couldn't get this in place for I/O Connect, but hope to have good news soon!. |
|
Could you extend to other LLMs in the model garden such as Llama 3? |
|
Yes... but... In the Vertex AI Model Garden, Llama 3 requires you to deploy the model to an endpoint you control. And the details of the API are poorly documented. Claude has more full API support with "model-as-a-service" and a standard endpoint. My current thinking is that I'll be able to provide direct support through the current classes we have (ie - you'll just be able to specify a claude model and it will work), while you may need to do some work for other models in the Model Garden, including deploying an endpoint. |
|
Hi @afirstenberg, Any update on the progress? Do you have an ETA. |
|
+1 is there any updates for this inquiry? |
|
+1 to Claude 3.5 Sonnet via Vertex |
|
It is still next on my list and in progress - but no updates to share. |
|
Looks like I'll be adding this for Mistral at the same time. |
|
Here's a temporary solution for using Anthropic with Vertex AI and LangChain. You need to install the following additional packages:
Your usage should look like this: import { GoogleAuth } from "google-auth-library";
import { ChatVertexAntropicAI } from "./ChatVertexAntropicAI";
const llm = new ChatVertexAntropicAI({
googleAuth: new GoogleAuth({
scopes: "https://www.googleapis.com/auth/cloud-platform",
// your credentials
}),
region: <region>,
projectId: <project_id>,
model: "claude-3-5-sonnet@20240620",
});
// Streaming
const response = await llm
.pipe(new StringOutputParser())
.stream([["human", "Hi, who are you?"]]);
for await (const chunk of response) {
console.log(chunk);
}
// None streaming
const response = await llm.invoke("Hi, who are you?");And here is the implementation for import {
SimpleChatModel,
type BaseChatModelParams,
} from "@langchain/core/language_models/chat_models";
import { CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";
import { AIMessageChunk, type BaseMessage } from "@langchain/core/messages";
import { ChatGenerationChunk } from "@langchain/core/outputs";
import { AnthropicVertex } from "@anthropic-ai/vertex-sdk";
import { GoogleAuth } from "google-auth-library";
export interface ChatVertexAntropicAIInput extends BaseChatModelParams {
googleAuth: GoogleAuth;
projectId: string;
region: string;
model: string;
maxTokens?: number;
temperature?: number;
}
export class ChatVertexAntropicAI extends SimpleChatModel {
client: AnthropicVertex;
model: string;
maxTokens: number;
temperature: number;
constructor(fields: ChatVertexAntropicAIInput) {
super(fields);
this.model = fields.model;
this.maxTokens = fields.maxTokens ?? 1000;
this.temperature = fields.temperature ?? 0;
this.client = new AnthropicVertex({
googleAuth: fields.googleAuth,
projectId: fields.projectId,
region: fields.region,
});
}
_llmType() {
return "vertex-ai-custom";
}
lg_to_anthropic(messages: BaseMessage[]) {
return messages.map((message) => {
const isUser = message.lc_id.pop() === "HumanMessage";
return {
role: isUser ? "user" : "assistant",
content: [
{
type: "text",
text: message.content,
},
],
};
}) as any;
}
async invoke_llm(messages: BaseMessage[]) {
const response = await this.client.messages.create({
model: this.model,
max_tokens: this.maxTokens,
temperature: this.temperature,
messages: this.lg_to_anthropic(messages),
});
return response.content[0].type === "text" ? response.content[0].text : "";
}
async stream_llm(messages: BaseMessage[]) {
const response = this.client.messages.stream({
model: this.model,
max_tokens: this.maxTokens,
temperature: this.temperature,
messages: this.lg_to_anthropic(messages),
});
return response;
}
async _call(
messages: BaseMessage[],
options: this["ParsedCallOptions"],
runManager?: CallbackManagerForLLMRun
): Promise<string> {
if (!messages.length) {
throw new Error("No messages provided.");
}
// Pass `runManager?.getChild()` when invoking internal runnables to enable tracing
// await subRunnable.invoke(params, runManager?.getChild());
if (typeof messages[0].content !== "string") {
throw new Error("Multimodal messages are not supported.");
}
return await this.invoke_llm(messages);
}
async *_streamResponseChunks(
messages: BaseMessage[],
options: this["ParsedCallOptions"],
runManager?: CallbackManagerForLLMRun
): AsyncGenerator<ChatGenerationChunk> {
if (!messages.length) {
throw new Error("No messages provided.");
}
if (typeof messages[0].content !== "string") {
throw new Error("Multimodal messages are not supported.");
}
// Pass `runManager?.getChild()` when invoking internal runnables to enable tracing
// await subRunnable.invoke(params, runManager?.getChild());
const response = await this.stream_llm(messages);
for await (const chunk of response) {
if (chunk.type === "content_block_delta") {
const delta = chunk.delta.type === "text_delta" ? chunk.delta.text : "";
yield new ChatGenerationChunk({
message: new AIMessageChunk({
content: delta,
}),
text: delta,
});
// Trigger the appropriate callback for new chunks
await runManager?.handleLLMNewToken(delta);
}
}
}
}Important NoteThis implementation is currently limited to handling text-only input and output. Other modalities and tool calling are not implemented. |
|
Edit: This is also easy to patch in @langchain/anthropic, a far better solution. @d-liya -- Nice approach, here is a more general one (should work with tools, only tested with react): import {BaseChatModel, type BaseChatModelParams, BindToolsInput} from "@langchain/core/language_models/chat_models";
import {AnthropicVertex} from "@anthropic-ai/vertex-sdk";
import {AIMessage, BaseMessage, BaseMessageChunk} from "@langchain/core/messages";
import {CallbackManagerForLLMRun} from "@langchain/core/callbacks/manager";
import {Runnable} from "@langchain/core/runnables";
import {BaseLanguageModelInput} from "@langchain/core/language_models/base";
import {convertToOpenAITool} from "@langchain/core/utils/function_calling";
import type {Tool as AnthropicTool} from "@anthropic-ai/sdk/resources/index.mjs";
export interface ChatAnthropicVertexInput extends BaseChatModelParams {
projectId: string;
region: string;
model: string;
maxTokens?: number;
temperature?: number;
topP?: number;
topK: number;
}
export function convertToAnthropicTool(tool: BindToolsInput): AnthropicTool {
const formatted = convertToOpenAITool(tool).function;
return {
name: formatted.name,
description: formatted.description,
input_schema: formatted.parameters as AnthropicTool.InputSchema
}
}
function formatMessagesAnthropic(messages: BaseMessage[]) {
let system_message = null;
let formatted_messages = [];
for (let [i, message] of messages.entries()) {
if (message._getType() === "system") {
if (i !== 0) throw new Error("system message must be first");
if (typeof message.content !== "string") throw new Error(`system message must be a string`);
system_message = message.content;
continue;
}
if (typeof message.content !== "string") {
throw new Error("not implemented");
} else if (message instanceof AIMessage) {
throw new Error("not implemented");
}
const role = message._getType() === "human" ? "user" : "assistant";
const content = message.content;
formatted_messages.push({role, content});
}
return [system_message, formatted_messages];
}
export class ChatAnthropicVertex extends BaseChatModel<ChatAnthropicVertexInput> {
client: AnthropicVertex;
model: string;
maxTokens: number;
temperature: number;
topP: number;
topK: number;
static lc_name(): string {
return "ChatAnthropicVertex";
}
get lc_secrets(): { [key: string]: string } {
return {
"authOptions.credentials": "GOOGLE_VERTEX_AI_WEB_CREDENTIALS",
};
}
_llmType(): string {
return "vertex-ai-custom";
}
_identifyingParams(): Record<string, any> {
return {
modelName: this.model
}
}
bindTools(tools: BindToolsInput[], kwargs?: Partial<ChatAnthropicVertexInput>): Runnable<BaseLanguageModelInput, BaseMessageChunk, ChatAnthropicVertexInput> {
const formattedTools = tools.map(convertToAnthropicTool);
return this.bind({
tools: formattedTools,
...kwargs
});
}
bind(kwargs: Partial<ChatAnthropicVertexInput>): Runnable<BaseLanguageModelInput, BaseMessageChunk, ChatAnthropicVertexInput> {
return super.bind(kwargs);
}
constructor(fields: ChatAnthropicVertexInput) {
super(fields);
this.model = fields.model;
this.maxTokens = fields.maxTokens ?? 1024;
this.temperature = fields.temperature ?? 0;
this.topP = fields.topP ?? 0.95;
this.topK = fields.topK ?? 40;
this.client = new AnthropicVertex({
projectId: fields.projectId,
region: fields.region,
});
}
async invokeLLM(messages: BaseMessage[], stop?: string[]) {
const [systemMessage, formattedMessages] = formatMessagesAnthropic(messages);
const response = await this.client.messages.create({
model: this.model,
max_tokens: this.maxTokens,
temperature: this.temperature,
top_p: this.topP,
top_k: this.topK,
messages: formattedMessages,
...systemMessage && {system: systemMessage},
...stop && {stop_sequences: stop}
})
return response.content[0].type === "text" ? response.content[0].text : "";
}
async _generate(
messages: BaseMessage[],
options: this["ParsedCallOptions"],
runManager?: CallbackManagerForLLMRun
) {
if (!messages.length) {
throw new Error("message length cant be 0");
}
if (typeof messages[0].content !== "string") {
throw new Error("cant work with multimodal messages yet");
}
const result = await this.invokeLLM(messages, options.stop);
return {
generations: [{message: new AIMessage(result), text: result}]
}
}
} |
|
Is it possible to modify import { expect, it } from 'vitest'
import { authenticate } from '../authenticate'
import { ChatAnthropic } from 'langchain-anthropic-edge'
import { AnthropicVertex } from '@anthropic-ai/vertex-sdk'
import { GoogleAuth } from 'google-auth-library'
import { AnthropicVertexWeb } from '../AnthropicVertex'
it('Call Anthropic Vertex on nodejs', async () => {
const client = new AnthropicVertex({
region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
googleAuth: new GoogleAuth({
credentials: {
client_email: import.meta.env
.VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
private_key: import.meta.env
.VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
},
scopes: ['https://www.googleapis.com/auth/cloud-platform'],
}),
})
const chat = new ChatAnthropic({
apiKey: 'test',
model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
createClient: (() => client) as any,
})
const response = await chat.invoke([['human', 'Hello!']])
console.log(response)
expect(response).not.undefined
})
it('Call Anthropic Vertex on edge runtime', async () => {
const token = await authenticate({
clientEmail: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_CLIENT_EMAIL!,
privateKey: import.meta.env.VITE_VERTEX_ANTROPIC_GOOGLE_SA_PRIVATE_KEY!,
})
const client = new AnthropicVertexWeb({
region: import.meta.env.VITE_VERTEX_ANTROPIC_REGION,
projectId: import.meta.env.VITE_VERTEX_ANTROPIC_PROJECTID,
accessToken: token.access_token,
})
const chat = new ChatAnthropic({
apiKey: 'test',
model: import.meta.env.VITE_VERTEX_ANTROPIC_MODEL,
createClient: (() => client) as any,
})
const response = await chat.invoke([['human', 'Hello!']])
console.log(response)
expect(response).not.undefined
}, 10_000) |
|
@rxliuli -- Yep, here is the diff we use to switch langchain js to AnthropicVertex. diff --git a/node_modules/@langchain/anthropic/dist/chat_models.js b/node_modules/@langchain/anthropic/dist/chat_models.js
index 14bbafc..46047e1 100644
--- a/node_modules/@langchain/anthropic/dist/chat_models.js
+++ b/node_modules/@langchain/anthropic/dist/chat_models.js
@@ -1,4 +1,3 @@
-import { Anthropic } from "@anthropic-ai/sdk";
import { AIMessageChunk } from "@langchain/core/messages";
import { ChatGenerationChunk } from "@langchain/core/outputs";
import { getEnvironmentVariable } from "@langchain/core/utils/env";
@@ -8,6 +7,7 @@ import { zodToJsonSchema } from "zod-to-json-schema";
import { RunnablePassthrough, RunnableSequence, } from "@langchain/core/runnables";
import { isZodSchema } from "@langchain/core/utils/types";
import { isLangChainTool } from "@langchain/core/utils/function_calling";
+import { AnthropicVertex } from "@anthropic-ai/vertex-sdk";
import { AnthropicToolsOutputParser } from "./output_parsers.js";
import { extractToolCallChunk, handleToolChoice } from "./utils/tools.js";
import { _formatMessagesForAnthropic } from "./utils/message_inputs.js";
@@ -422,6 +422,7 @@ export class ChatAnthropicMessages extends BaseChatModel {
return {
anthropicApiKey: "ANTHROPIC_API_KEY",
apiKey: "ANTHROPIC_API_KEY",
+ "authOptions.credentials": "GOOGLE_VERTEX_AI_WEB_CREDENTIALS",
};
}
get lc_aliases() {
@@ -537,11 +538,11 @@ export class ChatAnthropicMessages extends BaseChatModel {
});
this.anthropicApiKey =
fields?.apiKey ??
- fields?.anthropicApiKey ??
- getEnvironmentVariable("ANTHROPIC_API_KEY");
- if (!this.anthropicApiKey) {
- throw new Error("Anthropic API key not found");
- }
+ fields?.anthropicApiKey ??
+ getEnvironmentVariable("ANTHROPIC_API_KEY");
+ // if (!this.anthropicApiKey) {
+ // throw new Error("Anthropic API key not found");
+ // }
this.clientOptions = fields?.clientOptions ?? {};
/** Keep anthropicApiKey for backwards compatibility */
this.apiKey = this.anthropicApiKey;
@@ -755,10 +756,10 @@ export class ChatAnthropicMessages extends BaseChatModel {
async createStreamWithRetry(request, options) {
if (!this.streamingClient) {
const options_ = this.apiUrl ? { baseURL: this.apiUrl } : undefined;
- this.streamingClient = new Anthropic({
+ this.streamingClient = new AnthropicVertex({
...this.clientOptions,
...options_,
- apiKey: this.apiKey,
+ // apiKey: this.apiKey,
// Prefer LangChain built-in retries
maxRetries: 0,
});
@@ -774,13 +775,13 @@ export class ChatAnthropicMessages extends BaseChatModel {
async completionWithRetry(request, options) {
if (!this.batchClient) {
const options = this.apiUrl ? { baseURL: this.apiUrl } : undefined;
- if (!this.apiKey) {
- throw new Error("Missing Anthropic API key.");
- }
- this.batchClient = new Anthropic({
+ // if (!this.apiKey) {
+ // throw new Error("Missinxg Anthropic API key.");
+ // }
+ this.batchClient = new AnthropicVertex({
...this.clientOptions,
...options,
- apiKey: this.apiKey,
+ // apiKey: this.apiKey,
maxRetries: 0,
});
}
|
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
Description
I am trying to use ChatVertexAI with Anthropic Claude 3, but it seems this class only supports Gemini models and returns the above error message.
This appears to be a deliberate choice in the code:
langchainjs/libs/langchain-google-common/src/utils/common.ts
Lines 125 to 132 in b9d86b1
I've verified that using Claude and Vertex AI and Anthropics Vertex SDK works fine:
Output:
{ "id": "msg_vrtx_01M1yGR5LiteHznRmyK2MaPG", "type": "message", "role": "assistant", "model": "claude-3-opus-20240229", "stop_sequence": null, "usage": { "input_tokens": 10, "output_tokens": 12 }, "content": [ { "type": "text", "text": "Hello! How can I assist you today?" } ], "stop_reason": "end_turn" }System Info
Platform: Mac
Node: 20.11.0
The text was updated successfully, but these errors were encountered: