feat: added ibm watsonx chat model and streaming to base llm model #3577

faileon · 2023-12-07T09:11:00Z

Hi everyone,
as per the great PR #3399 based on the #2378 I added my implementation of ChatModel class and streaming capabilities to both base LLM model and the Chat Model.

I am not very familiar with turborepo, so I hope this structure is fine. Please let me know anything that should be changed.

vercel · 2023-12-07T09:11:05Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 13, 2023 1:22am
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		Dec 13, 2023 1:22am

dosubot · 2023-12-07T09:12:43Z

langchain/package.json

@@ -568,6 +568,9 @@
    "chat_models/fake.cjs",


Hey there! I've noticed that this PR introduces new dependencies related to chat models. This comment is just to flag the change for maintainers to review the impact on peer/dev/hard dependencies. Great work on the PR!

dosubot · 2023-12-07T09:12:44Z

langchain/src/chat_models/watsonx_ai.ts

@@ -0,0 +1,154 @@
+import { getEnvironmentVariable } from "@langchain/core/utils/env";


Hey team, I've reviewed the code changes and noticed that environment variables are being accessed and read using getEnvironmentVariable and process.env. I've flagged this for your attention to ensure the handling of environment variables aligns with best practices. Great work overall!

dosubot · 2023-12-07T09:12:46Z

langchain/src/llms/watsonx_ai.ts

@@ -1,49 +1,13 @@
-import { BaseLLMCallOptions, BaseLLMParams, LLM } from "./base.js";


Hey team, I've flagged this PR for your review as it appears to add, access, or require environment variables for configuring the Watsonx AI interaction. Please take a look to ensure that the handling of environment variables is appropriate and secure.

dosubot · 2023-12-07T09:12:47Z

langchain/src/util/watsonx-client.ts

@@ -0,0 +1,165 @@
+import {


Hey team, this PR introduces a new external HTTP request using the fetch API in the WatsonApiClient class to obtain an IAM token from the IBM IAM token endpoint. This comment is flagging the change for maintainers to review the addition of this new request. Great work on the implementation!

jacoblee93 · 2023-12-13T01:10:28Z

Thanks for your patience! @faileon @chasemcdo do you think you could have a last look? Moved around to account for recent refactoring

jacoblee93 · 2023-12-13T02:11:59Z

libs/langchain-community/src/chat_models/watsonx_ai.ts

+    });
+  }
+
+  protected _formatMessagesAsPrompt(messages: BaseMessage[]): string {


We should make a Llama adapter or something at some point...

Yes definitely, I was thinking about it too. Right now this is blatantly copy pasted from the Ollama model and assumes the user only uses llama based chat model. However Watson offers the ability to run many models, including spinning any model from HF...

jacoblee93 · 2023-12-13T02:13:17Z

libs/langchain-community/src/llms/watsonx_ai.ts


  constructor(fields: WatsonxAIParams) {
    super(fields);

-    this.region = fields?.region ?? this.region;


Isn't removing all of these a breaking change?

I moved a few attributes to the WatsonClient as it's something I also needed in the ChatModel. I did remove lc_serializable = true; by accident and will add it back in.

I tried running the provided example at examples/src/llms/watsonx_ai.ts and it works like before.

It is breaking in the sense that if someone is dependent on the old schema they'll need to update it, but most of the functionality is there, but in a now more reusable format.

One thing that I am noticing missing is there was originally an endpoint parameter which would allow the end user to overwrite the entire url if they so desired.

I had included this since the production endpoint is still labeled as "beta", so this would allow someone to migrate to a different (potentially more stable) endpoint as soon as it is released without the need to submit a PR to LangChain.

Thoughts on reintroducing that @faileon ?

Not sure what current usage is, but would love a shim for principle's sake. I can try to have a look later if you don't have time.

Ah I get it now, yes this would be breaking to someone using the old schema. I can revert it and make it backwards compatible.

Regarding the endpoint parameter that's a good catch, I agree it should be configurable, as it will most likely change once IBM releases non beta version.

I have some free time during this weekend again to work on it.

Thank you! This is a very recent integration, but we really try to not have any breaking changes in patches unless they are completely unavoidable.

chasemcdo

Looks awesome! Thanks for reverse engineering that

chasemcdo · 2023-12-13T15:30:05Z

libs/langchain-community/src/llms/watsonx_ai.ts

-    urlTokenParams.append(
-      "grant_type",
-      "urn:ibm:params:oauth:grant-type:apikey"
+  async *_streamResponseChunks(


I may be missing something, but it seems like while we have the stream function for the LLM it isn't actually usable in the current state?

I was expecting something closer to the sagemaker implementation which would allow easy toggling of streaming.

Though the present _call function appears to directly call generate_text.

I was originally inspired by the way the ollama connector handles it, but let me take another look at this. I only tested the chat model I implemented for streaming and invoking.

chasemcdo · 2023-12-13T15:39:58Z

libs/langchain-community/src/types/watsonx-types.ts

+  top_k?: number;
+  top_p?: number;
+  repetition_penalty?: number;
+}


I had originally left this undefined to avoid missing parameters, but looks like you've done a great job of adding them!

One thing to add might be the Random seed not sure what the expected argument is off the top of my head.

Good catch, I didn't see the Random seed. I will update the model.

chasemcdo · 2023-12-13T15:44:14Z

libs/langchain-community/src/llms/watsonx_ai.ts


  constructor(fields: WatsonxAIParams) {
    super(fields);

-    this.region = fields?.region ?? this.region;


It is breaking in the sense that if someone is dependent on the old schema they'll need to update it, but most of the functionality is there, but in a now more reusable format.

One thing that I am noticing missing is there was originally an endpoint parameter which would allow the end user to overwrite the entire url if they so desired.

I had included this since the production endpoint is still labeled as "beta", so this would allow someone to migrate to a different (potentially more stable) endpoint as soon as it is released without the need to submit a PR to LangChain.

Thoughts on reintroducing that @faileon ?

chasemcdo · 2023-12-14T00:13:02Z

libs/langchain-community/src/types/watsonx-types.ts

+export interface WatsonModelParameters {
+  decoding_method?: "sample" | "greedy";
+  max_new_tokens?: number;
+  min_new_tokens?: number;
+  stop_sequences?: string[];
+  temperature?: number;
+  top_k?: number;
+  top_p?: number;
+  repetition_penalty?: number;
+}


Did a bit more digging and looks like the Random Seed could be added like this:

Suggested change

export interface WatsonModelParameters {

decoding_method?: "sample" | "greedy";

max_new_tokens?: number;

min_new_tokens?: number;

stop_sequences?: string[];

temperature?: number;

top_k?: number;

top_p?: number;

repetition_penalty?: number;

}

export interface WatsonModelParameters {

decoding_method?: "sample" | "greedy";

max_new_tokens?: number;

min_new_tokens?: number;

stop_sequences?: string[];

temperature?: number;

top_k?: number;

top_p?: number;

repetition_penalty?: number;

random_seed?: number;

}

Awesome, thanks! I will update it.

bracesproul

thanks for this, code is looking pretty good! I left a few comments with some requests, once you've implemented please re-request my review

bracesproul · 2023-12-21T22:54:07Z

langchain/scripts/create-entrypoints.js

@@ -217,6 +217,7 @@ const entrypoints = {
  "chat_models/llama_cpp": "chat_models/llama_cpp",
  "chat_models/yandex": "chat_models/yandex",
  "chat_models/fake": "chat_models/fake",
+  "chat_models/watsonx_ai": "chat_models/watsonx_ai",


We no longer are adding integrations to the langchain package. Instead, only export inside langchain-community, from their create-entrypoints file and remove the re-exports from the main langchain package.
Thank you!

bracesproul · 2023-12-21T22:55:55Z

langchain/src/chat_models/watsonx_ai.ts

@@ -0,0 +1 @@
+export * from "@langchain/community/chat_models/watsonx_ai";


bracesproul · 2023-12-21T22:58:58Z

libs/langchain-community/src/chat_models/watsonx_ai.ts

+export class WatsonxAIChat extends SimpleChatModel {
+  private readonly watsonApiClient: WatsonApiClient;
+
+  readonly modelId!: string;


Don't use definite assignment assertions, instead make it a non nullable type and add checks in the constructor that verify the value is defined.

These can come back to bite you in production (I've done this before and it was not fun to debug 😅)

bracesproul · 2023-12-21T23:01:56Z

libs/langchain-community/src/chat_models/watsonx_ai.ts

+
+  readonly projectId!: string;
+
+  constructor(fields: WatsonxAIParams & BaseChatModelParams) {


Lets redefine this as an interface above the class:

interface WatsonxAIChatParams extends WatsonxAIParams, BaseChatModelParams {};

This way it's easy to:

export the class params type for use outside this file

add extra properties in the future.

Also, I believe all the properties on those two interfaces are optional, so we should be able to do:

Suggested change

constructor(fields: WatsonxAIParams & BaseChatModelParams) {

constructor(fields?: WatsonxAIChatParams) {

bracesproul · 2023-12-21T23:10:53Z

libs/langchain-community/src/chat_models/watsonx_ai.ts

+    return chunks.join("");
+  }
+
+  override async *_streamResponseChunks(


I don't think you need the override here

bracesproul · 2023-12-21T23:11:45Z

libs/langchain-community/src/chat_models/watsonx_ai.ts

+    _messages: BaseMessage[],
+    _options: this["ParsedCallOptions"],
+    _runManager?: CallbackManagerForLLMRun


Drop the underscore if they're being used. Typically, variables prefixed with an underscore are unused, and the underscore is used to bypass a lint rule for no-unused-variables

bracesproul · 2023-12-21T23:16:49Z

libs/langchain-community/src/llms/watsonx_ai.ts

- * The WatsonxAIParams interface defines the input parameters for
- * the WatsonxAI class.
- */
-export interface WatsonxAIParams extends BaseLLMParams {


If you move this interface declaration you'll need to re-export it from this file to keep backwards compatibility

bracesproul · 2023-12-21T23:18:21Z

libs/langchain-community/src/utils/watsonx-client.ts

+export class WatsonApiClient {
+  private iamToken?: IAMTokenResponse;
+
+  private readonly apiKey!: string;


Same as comment above, no definite assignment assertions

bracesproul · 2023-12-21T23:19:33Z

libs/langchain-community/src/utils/watsonx-client.ts

+    const formBody = Object.entries(payload).reduce((acc, [key, value]) => {
+      const encodedKey = encodeURIComponent(key as string);
+      const encodedValue = encodeURIComponent(value as string);
+      acc.push(`${encodedKey}=${encodedValue}`);
+      return acc;
+    }, [] as string[]);
+    const body = formBody.join("&");


Could this be better handled with new URLSearchParams instead?

streaming, chat model

ed8f8ff

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Dec 7, 2023

dosubot bot added the auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features label Dec 7, 2023

dosubot bot reviewed Dec 7, 2023

View reviewed changes

vercel bot deployed to Preview – langchainjs-api-refs December 7, 2023 09:17 View deployment

vercel bot deployed to Preview – langchainjs-docs December 7, 2023 09:18 View deployment

jacoblee93 added 4 commits December 12, 2023 16:56

Merge

ac4a850

Format

8a52a65

Remove unused files

48e4380

Rename

f6dba28

jacoblee93 added question Further information is requested lgtm PRs that are ready to be merged as-is labels Dec 13, 2023

vercel bot deployed to Preview – langchainjs-docs December 13, 2023 01:16 View deployment

vercel bot deployed to Preview – langchainjs-api-refs December 13, 2023 01:22 View deployment

jacoblee93 reviewed Dec 13, 2023

View reviewed changes

jacoblee93 added close PRs that need one or two touch-ups to be ready and removed lgtm PRs that are ready to be merged as-is labels Dec 13, 2023

chasemcdo reviewed Dec 13, 2023

View reviewed changes

chasemcdo reviewed Dec 14, 2023

View reviewed changes

bracesproul reviewed Dec 21, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: added ibm watsonx chat model and streaming to base llm model #3577

feat: added ibm watsonx chat model and streaming to base llm model #3577

faileon commented Dec 7, 2023

vercel bot commented Dec 7, 2023 •

edited

Loading

dosubot bot Dec 7, 2023

dosubot bot Dec 7, 2023

dosubot bot Dec 7, 2023

dosubot bot Dec 7, 2023

jacoblee93 commented Dec 13, 2023

jacoblee93 Dec 13, 2023 •

edited

Loading

faileon Dec 13, 2023

jacoblee93 Dec 13, 2023

faileon Dec 13, 2023

chasemcdo Dec 13, 2023

jacoblee93 Dec 13, 2023 •

edited

Loading

faileon Dec 14, 2023

jacoblee93 Dec 14, 2023 •

edited

Loading

chasemcdo left a comment

chasemcdo Dec 13, 2023

faileon Dec 14, 2023

chasemcdo Dec 13, 2023

faileon Dec 14, 2023

chasemcdo Dec 13, 2023

chasemcdo Dec 14, 2023

faileon Dec 14, 2023

bracesproul left a comment

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

bracesproul Dec 21, 2023

		@@ -0,0 +1,154 @@
		import { getEnvironmentVariable } from "@langchain/core/utils/env";

		@@ -1,49 +1,13 @@
		import { BaseLLMCallOptions, BaseLLMParams, LLM } from "./base.js";

		@@ -0,0 +1 @@
		export * from "@langchain/community/chat_models/watsonx_ai";


		readonly projectId!: string;

		constructor(fields: WatsonxAIParams & BaseChatModelParams) {

	constructor(fields: WatsonxAIParams & BaseChatModelParams) {
	constructor(fields?: WatsonxAIChatParams) {

feat: added ibm watsonx chat model and streaming to base llm model #3577

Are you sure you want to change the base?

feat: added ibm watsonx chat model and streaming to base llm model #3577

Conversation

faileon commented Dec 7, 2023

vercel bot commented Dec 7, 2023 • edited Loading

dosubot bot Dec 7, 2023

Choose a reason for hiding this comment

dosubot bot Dec 7, 2023

Choose a reason for hiding this comment

dosubot bot Dec 7, 2023

Choose a reason for hiding this comment

dosubot bot Dec 7, 2023

Choose a reason for hiding this comment

jacoblee93 commented Dec 13, 2023

jacoblee93 Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacoblee93 Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacoblee93 Dec 14, 2023 • edited Loading

Choose a reason for hiding this comment

chasemcdo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bracesproul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Dec 7, 2023 •

edited

Loading

jacoblee93 Dec 13, 2023 •

edited

Loading

jacoblee93 Dec 13, 2023 •

edited

Loading

jacoblee93 Dec 14, 2023 •

edited

Loading