Merge branch 'main' into mass/discord-api-integration

.github/contributing/INTEGRATIONS.md

@@ -152,7 +152,7 @@ Below are links to guides with advice and tips for specific types of integration
 - [Vector stores](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/VECTOR_STORES.md) (e.g. Pinecone)
 - [Persistent message stores](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/MESSAGE_STORES.md) (used to persistently store and load raw chat histories, e.g. Redis)
 - [Document loaders](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/DOCUMENT_LOADERS.md) (used to load documents for later storage into vector stores, e.g. Apify)
-- Embeddings (TODO) (e.g. Cohere)
+- [Embeddings](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/EMBEDDINGS.md) (used to create embeddings of text documents or strings e.g. Cohere)
 - [Tools](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/TOOLS.md) (used for agents, e.g. the SERP API tool)
 
 This is a living document, so please make a pull request if we're missing anything useful!

.github/contributing/integrations/EMBEDDINGS.md

@@ -0,0 +1,21 @@
+# Contributing third-party Text Embeddings
+
+This page contains some specific guidelines and examples for contributing integrations with third-party Text Embedding providers.
+
+**Make sure you read the [general guidelines page](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/INTEGRATIONS.md) first!**
+
+## Example PR
+
+We'll be referencing this PR adding Gradient Embeddings as an example: https://github.com/langchain-ai/langchainjs/pull/3475
+
+## General ideas
+
+The general idea for adding new third-party Text Embeddings is to subclass the `Embeddings` class and implement the `embedDocuments` and `embedQuery` methods. 
+
+The `embedDocuments` method should take a list of documents and return a list of embeddings for each document. The `embedQuery` method should take a query and return an embedding for that query.
+
+`embedQuery` can typically be implemented by calling `embedDocuments` with a list containing only the query.
+
+## Wrap Text Embeddings requests in this.caller
+
+The base Embeddings class contains an instance property called `caller` that will automatically handle retries, errors, timeouts, and more. You should wrap calls to the LLM in `this.caller.call` [as shown here](https://github.com/langchain-ai/langchainjs/blob/f469ec00d945a3f8421b32f4be78bce3f66a74bb/langchain/src/embeddings/gradient_ai.ts#L72)

CONTRIBUTING.md

@@ -114,6 +114,14 @@ To get started, you will need to install the dependencies for the project. To do
 yarn
 ```
 
+Then, you will need to switch directories into `langchain-core` and build core by running:
+
+```bash
+cd ../langchain-core
+yarn
+yarn build
+```
+
 ### Linting
 
 We use [eslint](https://eslint.org/) to enforce standard lint rules.
@@ -230,7 +238,7 @@ example to showcase how to use it. Most of our users find examples to be the
 most helpful kind of documentation.
 
 Examples can be added in the `examples/src` directory, e.g.
-`examples/src/path/to/example` and should export a `run` function. This
+`examples/src/path/to/example`. This
 example can then be invoked with `yarn example path/to/example` at the top
 level of the repo.
 

docs/api_refs/typedoc.json

@@ -24,6 +24,7 @@
     "./langchain/src/agents/load.ts",
     "./langchain/src/agents/toolkits/index.ts",
     "./langchain/src/agents/toolkits/aws_sfn.ts",
+    "./langchain/src/agents/toolkits/connery/index.ts",
     "./langchain/src/agents/toolkits/sql/index.ts",
     "./langchain/src/agents/format_scratchpad/openai_functions.ts",
     "./langchain/src/agents/format_scratchpad/openai_tools.ts",
@@ -39,9 +40,11 @@
     "./langchain/src/tools/aws_sfn.ts",
     "./langchain/src/tools/calculator.ts",
     "./langchain/src/tools/discord.ts",
+    "./langchain/src/tools/connery.ts",
     "./langchain/src/tools/render.ts",
     "./langchain/src/tools/sql.ts",
     "./langchain/src/tools/webbrowser.ts",
+    "./langchain/src/tools/gmail/index.ts",
     "./langchain/src/tools/google_calendar/index.ts",
     "./langchain/src/tools/google_places.ts",
     "./langchain/src/chains/index.ts",
@@ -68,6 +71,7 @@
     "./langchain/src/embeddings/minimax.ts",
     "./langchain/src/embeddings/voyage.ts",
     "./langchain/src/embeddings/llama_cpp.ts",
+    "./langchain/src/embeddings/gradient_ai.ts",
     "./langchain/src/llms/load.ts",
     "./langchain/src/llms/base.ts",
     "./langchain/src/llms/openai.ts",
@@ -175,6 +179,7 @@
     "./langchain/src/document_loaders/fs/epub.ts",
     "./langchain/src/document_loaders/fs/csv.ts",
     "./langchain/src/document_loaders/fs/notion.ts",
+    "./langchain/src/document_loaders/fs/obsidian.ts",
     "./langchain/src/document_loaders/fs/unstructured.ts",
     "./langchain/src/document_loaders/fs/openai_whisper_audio.ts",
     "./langchain/src/document_loaders/fs/pptx.ts",

docs/core_docs/docs/integrations/llms/gradient_ai.mdx

@@ -32,6 +32,15 @@ const model = new GradientLLM({
 ## Usage
 
 import CodeBlock from "@theme/CodeBlock";
-import GradientLLMExample from "@examples/llms/gradient_ai.ts";
+import GradientLLMBaseExample from "@examples/llms/gradient_ai-base.ts";
+import GradientLLMAdapterExample from "@examples/llms/gradient_ai-adapter.ts";
 
-<CodeBlock language="typescript">{GradientLLMExample}</CodeBlock>
+### Using Gradient's Base Models
+
+<CodeBlock language="typescript">{GradientLLMBaseExample}</CodeBlock>
+
+### Using your own fine-tuned Adapters
+
+The use your own custom adapter simply set `adapterId` during setup.
+
+<CodeBlock language="typescript">{GradientLLMAdapterExample}</CodeBlock>

docs/core_docs/docs/integrations/text_embedding/gradient_ai.mdx

@@ -0,0 +1,39 @@
+---
+sidebar_class_name: node-only
+---
+
+# Gradient AI
+
+The `GradientEmbeddings` class uses the Gradient AI API to generate embeddings for a given text.
+
+## Setup
+
+You'll need to install the official Gradient Node SDK as a peer dependency:
+
+```bash npm2yarn
+npm i @gradientai/nodejs-sdk
+```
+
+You will need to set the following environment variables for using the Gradient AI API.
+
+```
+export GRADIENT_ACCESS_TOKEN=<YOUR_ACCESS_TOKEN>
+export GRADIENT_WORKSPACE_ID=<YOUR_WORKSPACE_ID>
+```
+
+Alternatively, these can be set during the GradientAI Class instantiation as `gradientAccessKey` and `workspaceId` respectively.
+For example:
+
+```typescript
+const model = new GradientEmbeddings({
+  gradientAccessKey: "My secret Access Token"
+  workspaceId: "My secret workspace id"
+});
+```
+
+## Usage
+
+import CodeBlock from "@theme/CodeBlock";
+import GradientEmbeddingsExample from "@examples/embeddings/gradient_ai.ts";
+
+<CodeBlock language="typescript">{GradientEmbeddingsExample}</CodeBlock>

docs/core_docs/docs/integrations/toolkits/connery.mdx

@@ -0,0 +1,25 @@
+import CodeBlock from "@theme/CodeBlock";
+import Example from "@examples/agents/connery_mrkl.ts";
+
+# Connery Actions Toolkit
+
+Using this toolkit, you can integrate Connery actions into your LangChain agents and chains.
+
+## What is Connery?
+
+Connery is an open-source plugin infrastructure for AI.
+
+With Connery, you can easily create a custom plugin, which is essentially a set of actions, and use them in your LangChain agents and chains.
+Connery will handle the rest: runtime, authorization, secret management, access management, audit logs, and other vital features.
+Also, you can find a lot of ready-to-use plugins from our community.
+
+Learn more about Connery:
+
+- GitHub repository: https://github.com/connery-io/connery-platform
+- Documentation: https://docs.connery.io
+
+## Usage
+
+This example shows how to create an agent with Connery actions using the Connery Actions Toolkit.
+
+<CodeBlock language="typescript">{Example}</CodeBlock>

docs/core_docs/docs/integrations/tools/connery.mdx

@@ -0,0 +1,25 @@
+import CodeBlock from "@theme/CodeBlock";
+import Example from "@examples/tools/connery.ts";
+
+# Connery Actions Tool
+
+Using this tool, you can integrate individual Connery actions into your LangChain agents and chains.
+
+## What is Connery?
+
+Connery is an open-source plugin infrastructure for AI.
+
+With Connery, you can easily create a custom plugin, which is essentially a set of actions, and use them in your LangChain agents and chains.
+Connery will handle the rest: runtime, authorization, secret management, access management, audit logs, and other vital features.
+Also, you can find a lot of ready-to-use plugins from our community.
+
+Learn more about Connery:
+
+- GitHub repository: https://github.com/connery-io/connery-platform
+- Documentation: https://docs.connery.io
+
+## Usage
+
+This example shows how to create a tool for one specific Connery action and call it.
+
+<CodeBlock language="typescript">{Example}</CodeBlock>

docs/core_docs/docs/integrations/tools/gmail.mdx

@@ -0,0 +1,27 @@
+---
+hide_table_of_contents: true
+---
+
+import CodeBlock from "@theme/CodeBlock";
+
+# Gmail Tool
+
+The Gmail Tool allows your agent to create and view messages from a linked email account.
+
+## Setup
+
+You will need to get an API key from [Google here](https://developers.google.com/gmail/api/guides)
+and [enable the new Gmail API](https://console.cloud.google.com/apis/library/gmail.googleapis.com).
+Then, set the environment variables for `GMAIL_CLIENT_EMAIL`, and either `GMAIL_PRIVATE_KEY`, or `GMAIL_KEYFILE`.
+
+To use the Gmail Tool you need to install the following official peer dependency:
+
+```bash npm2yarn
+npm install googleapis
+```
+
+## Usage
+
+import ToolExample from "@examples/tools/gmail.ts";
+
+<CodeBlock language="typescript">{ToolExample}</CodeBlock>

docs/core_docs/docs/integrations/vectorstores/xata.mdx

@@ -17,7 +17,7 @@ npm install @xata.io/cli -g
 In the [Xata UI](https://app.xata.io) create a new database. You can name it whatever you want, but for this example we'll use `langchain`.
 Create a table, again you can name it anything, but we will use `vectors`. Add the following columns via the UI:
 
-- `content` of type "Long text". This is used to store the `Document.pageContent` values.
+- `content` of type "Text". This is used to store the `Document.pageContent` values.
 - `embedding` of type "Vector". Use the dimension used by the model you plan to use (1536 for OpenAI).
 - any other columns you want to use as metadata. They are populated from the `Document.metadata` object. For example, if in the `Document.metadata` object you have a `title` property, you can create a `title` column in the table and it will be populated.
 

docs/core_docs/docs/modules/agents/agent_types/openai_functions_agent.mdx

@@ -55,10 +55,10 @@ Then, update your prompt to include another `MessagesPlaceholder`. This time we'
 
 ```typescript
 const prompt = ChatPromptTemplate.fromMessages([
-  ["ai", "You are a helpful assistant"],
+  ["ai", "You are a helpful assistant."],
+  new MessagesPlaceholder("chat_history"),
   ["human", "{input}"],
   new MessagesPlaceholder("agent_scratchpad"),
-  new MessagesPlaceholder("chat_history"),
 ]);
 ```
 
@@ -77,9 +77,14 @@ const runnableAgent = RunnableSequence.from([
     },
   },
   prompt,
-  modelWithTools,
+  modelWithFunctions,
   new OpenAIFunctionsAgentOutputParser(),
 ]);
+
+const executor = AgentExecutor.fromAgentAndTools({
+  agent: runnableAgent,
+  tools,
+});
 ```
 
 Finally we can call the agent, and save the output after the response is returned.
@@ -122,8 +127,8 @@ console.log(result2);
 
 You may also inspect the LangSmith traces for both agent calls here:
 
-- [Question 1](https://smith.langchain.com/public/12f4022d-f02f-4b00-9122-8493afe5507d/r)
-- [Question 2](https://smith.langchain.com/public/2052d0d1-5640-44bc-9024-d443d46db2a4/r)
+- [Question 1](https://smith.langchain.com/public/c1136951-f3f0-4ff5-a862-8db5d6bc8d04/r)
+- [Question 2](https://smith.langchain.com/public/b536cdc0-9bc9-4bdf-9298-4d6d7f88556b/r)
 
 # With `initializeAgentExecutorWithOptions`
 

docs/core_docs/docs/modules/agents/agent_types/openai_tools_agent.mdx

@@ -65,10 +65,10 @@ Then, update your prompt to include another `MessagesPlaceholder`. This time we'
 
 ```typescript
 const prompt = ChatPromptTemplate.fromMessages([
-  ["ai", "You are a helpful assistant"],
+  ["ai", "You are a helpful assistant."],
+  new MessagesPlaceholder("chat_history"),
   ["human", "{input}"],
   new MessagesPlaceholder("agent_scratchpad"),
-  new MessagesPlaceholder("chat_history"),
 ]);
 ```
 

docs/core_docs/docs/modules/agents/agent_types/structured_chat.mdx

@@ -1,5 +1,10 @@
 # Structured tool chat
 
+:::info
+If you are using a functions-capable model like ChatOpenAI, we currently recommend that you use the [OpenAI Functions agent](/docs/modules/agents/agent_types/openai_functions_agent)
+for more complex tool calling.
+:::
+
 The structured tool chat agent is capable of using multi-input tools.
 
 Older agents are configured to specify an action input as a single string, but this agent can use the provided tools' `args_schema` to populate the action input.

docs/core_docs/docs/modules/agents/how_to/streaming.mdx

@@ -0,0 +1,15 @@
+import CodeBlock from "@theme/CodeBlock";
+import StreamingExample from "@examples/agents/stream.ts";
+
+# Streaming
+
+Agents have the ability to stream iterations and actions back while they're still working.
+This can be very useful for any realtime application where you have users who need insights on the agent's progress while is has yet to finish.
+
+Setting up streaming with agents is very simple, even with existing agents. The only change required is switching your `executor.invoke({})` to be `executor.stream({})`.
+
+Below is a simple example of streaming with an agent.
+
+You can find the [LangSmith](https://smith.langchain.com/) trace for this example by clicking [here](https://smith.langchain.com/public/08978fa7-bb99-427b-850e-35773cae1453/r).
+
+<CodeBlock language="typescript">{StreamingExample}</CodeBlock>

docs/core_docs/docs/modules/callbacks/how_to/with_listeners.mdx

@@ -0,0 +1,19 @@
+import CodeBlock from "@theme/CodeBlock";
+import Example from "@examples/guides/expression_language/with_listeners.ts";
+
+# Listeners
+
+LangChain callbacks offer a method `withListeners` which allow you to add event listeners to the following events:
+
+- `onStart` - called when the chain starts
+- `onEnd` - called when the chain ends
+- `onError` - called when an error occurs
+
+These methods accept a callback function which will be called when the event occurs. The callback function can accept two arguments:
+
+- `input` - the input value, for example it would be `RunInput` if used with a Runnable.
+- `config` - an optional config object. This can contain metadata, callbacks or any other values passed in as a config object when the chain is started.
+
+Below is an example which demonstrates how to use the `withListeners` method:
+
+<CodeBlock language="typescript">{Example}</CodeBlock>