Update README and contributing guidelines

.github/contributing/INTEGRATIONS.md

@@ -10,7 +10,7 @@ The following guidelines apply broadly to all type of integrations:
 
 ### Creating a separate entrypoint
 
-You should generally not export your new module from an `index.ts` file that contains many other exports. Instead, you should add a separate entrypoint for your integration in [`langchain/scripts/create-entrypoints.js`](https://github.com/langchain-ai/langchainjs/blob/main/langchain/scripts/create-entrypoints.js) within the `entrypoints` object:
+You should generally not export your new module from an `index.ts` file that contains many other exports. Instead, you should add a separate entrypoint for your integration in [`libs/langchain-community/scripts/create-entrypoints.js`](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/scripts/create-entrypoints.js) within the `entrypoints` object:
 
 ```js
 import * as fs from "fs";
@@ -55,17 +55,17 @@ const entrypoints = {
 };
 ```
 
-A user would then import your new vector store as `import { LangCoVectorStore } from "langchain/vectorstores/langco";`.
+A user would then import your new vector store as `import { LangCoVectorStore } from "@langchain/community/vectorstores/langco";`.
 
 ### Third-party dependencies
 
-You may use third-party dependencies in new integrations, but they should be added as `peerDependencies` and `devDependencies` with an entry under `peerDependenciesMeta` in [`langchain/package.json`](https://github.com/langchain-ai/langchainjs/blob/main/langchain/package.json), **not under any core `dependencies` list**. This keeps the overall package size small, as only people who are using your integration will need to install, and allows us to support a wider range of runtimes.
+You may use third-party dependencies in new integrations, but they should be added as `peerDependencies` and `devDependencies` with an entry under `peerDependenciesMeta` in [`libs/langchain-community/package.json`](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/package.json), **not under any core `dependencies` list**. This keeps the overall package size small, as only people who are using your integration will need to install, and allows us to support a wider range of runtimes.
 
 We suggest using caret syntax (`^`) for peer dependencies to support a wider range of people trying to use them as well as to be somewhat tolerant to non-major version updates, which should (theoretically) be the only breaking ones.
 
 Please make sure all introduced dependencies are permissively licensed (MIT is recommended) and well-supported and maintained.
 
-You must also add your new entrypoint under `requiresOptionalDependency` in the [`create-entrypoints.js`](https://github.com/langchain-ai/langchainjs/blob/main/langchain/scripts/create-entrypoints.js) file to avoid breaking the build:
+You must also add your new entrypoint under `requiresOptionalDependency` in the [`create-entrypoints.js`](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/scripts/create-entrypoints.js) file to avoid breaking the build:
 
 ```js
 // Entrypoints in this list require an optional dependency to be installed.
@@ -142,9 +142,23 @@ This allows the linter and formatter to pick up example code blocks within docs
 
 As with all contributions, make sure you run `yarn lint` and `yarn format` so that everything conforms to our established style.
 
+### Separate integration packages
+
+While most integrations should generally reside in the `libs/langchain-community` workspace and be imported as `@langchain/community/module/name`, more in-depth integrations or suites of integrations may also reside in separate packages that depend on and extend `@langchain/core`. See [`@langchain/google-genai`](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-genai) for an example.
+
+To make creating packages like this easier, we offer the [`create-langchain-integration`](https://github.com/langchain-ai/langchainjs/blob/main/libs/create-langchain-integration/) utility that will automatically scaffold a repo with support for both ESM + CJS entrypoints. You can run it like this:
+
+```bash
+$ npx create-langchain-integration
+```
+
+The workflows and considerations for these packages are mostly the same as those in `@langchain/community`, with the exception that third-party dependencies should be hard dependencies instead of peer dependencies since the end-user will manually install your integration package anyway.
+
+You will need to make sure that your package is compatible with the current minor version of `@langchain/core` in order for it to be interoperable with other integration packages and the latest versions of LangChain. We recommend using a tilde syntax for your integration package's `@langchain/core` dependency to support a wider range of core patch versions.
+
 ## Integration-specific guidelines and example PRs
 
-Below are links to guides with advice and tips for specific types of integrations:
+Below are links to guides with advice and tips for specific types of integrations. These are currently out of date with the `@langchain/community` split, but will give you a rough idea of what is necessary:
 
 - [LLM providers](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/integrations/LLMS.md) (e.g. OpenAI's GPT-3)
 - Chat model providers (TODO) (e.g. Anthropic's Claude, OpenAI's GPT-4)

CONTRIBUTING.md

@@ -17,14 +17,22 @@ If you are not sure what to work on, we have a few suggestions:
 
 ### New abstractions
 
-We aim to keep the same APIs between the Python and JS versions of LangChain, where possible. As such we ask that if you have an idea for a new abstraction, please open an issue first to discuss it. This will help us make sure that the API is consistent across both versions. If you're not sure what to work on, we recommend looking at the links above first.
+We aim to keep the same core APIs between the Python and JS versions of LangChain, where possible. As such we ask that if you have an idea for a new abstraction, please open an issue first to discuss it. This will help us make sure that the API is consistent across both versions. If you're not sure what to work on, we recommend looking at the links above first.
 
 ### Want to add a specific integration?
 
 LangChain supports several different types of integrations with third-party providers and frameworks, including LLM providers (e.g. [OpenAI](https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/llms/openai.ts)), vector stores (e.g. [FAISS](https://github.com/ewfian/langchainjs/blob/main/langchain/src/vectorstores/faiss.ts)), document loaders (e.g. [Apify](https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/document_loaders/web/apify_dataset.ts)) persistent message history stores (e.g. [Redis](https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/stores/message/redis.ts)), and more.
 
 We welcome such contributions, but ask that you read our dedicated [integration contribution guide](https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/INTEGRATIONS.md) for specific details and patterns to consider before opening a pull request.
 
+These should generally reside in the `libs/langchain-community` workspace and be imported as `@langchain/community/module/name`, but more in-depth integrations or suites of integrations may also reside in separate packages that depend on and extend `@langchain/core`. See [`@langchain/google-genai`](https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-google-genai) for an example.
+
+To make creating packages like this easier, we offer the [`create-langchain-integration`](https://github.com/langchain-ai/langchainjs/blob/main/libs/create-langchain-integration/) utility that will automatically scaffold a repo with support for both ESM + CJS entrypoints. You can run it like this:
+
+```bash
+$ npx create-langchain-integration
+```
+
 ### Want to add a feature that's already in Python?
 
 If you're interested in contributing a feature that's already in the [LangChain Python repo](https://github.com/langchain-ai/langchain) and you'd like some help getting started, you can try pasting code snippets and classes into the [LangChain Python to JS translator](https://langchain-translator.vercel.app/).
@@ -48,7 +56,7 @@ If you are adding an issue, please try to keep it focused on a single modular bu
 If the two issues are related, or blocking, please link them rather than keep them as one single one.
 
 We will try to keep these issues as up to date as possible, though
-with the rapid rate of develop in this field some may get out of date.
+with the rapid rate of development in this field some may get out of date.
 If you notice this happening, please just let us know.
 
 ### 🙋 Getting Help
@@ -100,12 +108,18 @@ Next, try running the following common tasks:
 ## ✅ Common Tasks
 
 Our goal is to make it as easy as possible for you to contribute to this project.
-All of the below commands should be run from within the `langchain/` directory unless otherwise noted.
+All of the below commands should be run from within a workspace directory (e.g. `langchain`, `libs/langchain-community`) unless otherwise noted.
 
 ```bash
 cd langchain
 ```
 
+Or, if you are working on a community integration:
+
+```bash
+cd libs/langchain-community
+```
+
 ### Setup
 
 To get started, you will need to install the dependencies for the project. To do so, run:
@@ -164,7 +178,7 @@ yarn test
 
 #### Running a single test
 
-To run a single test, run:
+To run a single test, run the following from within a workspace:
 
 ```bash
 yarn test:single /path/to/yourtest.test.ts
@@ -177,7 +191,7 @@ This is useful for developing individual features.
 If you add support for a new external API, please add a new integration test.
 Integration tests should be called `*.int.test.ts`.
 
-Note that most integration tests require credentials or other setup. You will likely need to set up a `langchain/.env` file
+Note that most integration tests require credentials or other setup. You will likely need to set up a `langchain/.env` or `libs/langchain-community/.env` file
 like the example [here](https://github.com/langchain-ai/langchainjs/blob/main/langchain/.env.example).
 
 We generally recommend only running integration tests with `yarn test:single`, but if you want to run all integration tests, run:
@@ -205,7 +219,7 @@ import { OpenAI } from "langchain/llms/openai";
 We call these subpaths "entrypoints". In general, you should create a new entrypoint if you are adding a new integration with a 3rd party library. If you're adding self-contained functionality without any external dependencies, you can add it to an existing entrypoint.
 
 In order to declare a new entrypoint that users can import from, you
-should edit the `langchain/scripts/create-entrypoints.js` script. To add an
+should edit the `langchain/scripts/create-entrypoints.js` or `libs/langchain-community/scripts/create-entrypoints.js` script. To add an
 entrypoint `tools` that imports from `tools/index.ts` you'd add
 the following to the `entrypoints` variable:
 
@@ -242,7 +256,7 @@ Examples can be added in the `examples/src` directory, e.g.
 example can then be invoked with `yarn example path/to/example` at the top
 level of the repo.
 
-To run examples that require an environment variable, you'll need to add a `.env` file under `examples/.env`
+To run examples that require an environment variable, you'll need to add a `.env` file under `examples/.env`.
 
 ### Build Documentation Locally
 

README.md

@@ -32,9 +32,26 @@ LangChain is written in TypeScript and can be used in:
 - Browser
 - Deno
 
-## 🤔 What is this?
+## 🤔 What is LangChain?
 
-Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. However, using these LLMs in isolation is often insufficient for creating a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.
+**LangChain** is a framework for developing applications powered by language models. It enables applications that:
+- **Are context-aware**: connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.)
+- **Reason**: rely on a language model to reason (about how to answer based on provided context, what actions to take, etc.)
+
+This framework consists of several parts.
+- **LangChain Libraries**: The Python and JavaScript libraries. Contains interfaces and integrations for a myriad of components, a basic run time for combining these components into chains and agents, and off-the-shelf implementations of chains and agents.
+- **[LangChain Templates](https://github.com/langchain-ai/langchain/tree/master/templates)**: (currently Python-only) A collection of easily deployable reference architectures for a wide variety of tasks.
+- **[LangServe](https://github.com/langchain-ai/langserve)**: (currently Python-only) A library for deploying LangChain chains as a REST API.
+- **[LangSmith](https://smith.langchain.com)**: A developer platform that lets you debug, test, evaluate, and monitor chains built on any LLM framework and seamlessly integrates with LangChain.
+
+The LangChain libraries themselves are made up of several different packages.
+- **[`@langchain/core`](langchain-core)**: Base abstractions and LangChain Expression Language.
+- **[`@langchain/community`](libs/langchain-community)**: Third party integrations.
+- **[`langchain`](langchain)**: Chains, agents, and retrieval strategies that make up an application's cognitive architecture.
+
+Integrations may also be split into their own compatible packages.
+
+![LangChain Stack](docs/core_docs/static/img/langchain_stack.png)
 
 This library aims to assist in the development of those types of applications. Common examples of these applications include:
 
@@ -49,59 +66,46 @@ This library aims to assist in the development of those types of applications. C
 - [Documentation](https://js.langchain.com/docs/modules/model_io/models/chat/)
 - End-to-end Example: [Chat-LangChain](https://github.com/langchain-ai/chat-langchain)
 
-## 📖 Documentation
-
-Please see [here](https://js.langchain.com/docs/) for full documentation on:
-
-- Getting started (installation, setting up the environment, simple examples)
-- How-To examples (demos, integrations, helper functions)
-- Reference (full API docs)
-- Resources (high-level explanation of core concepts)
-
-
-## 🖇️ Relationship with Python LangChain
-
-This is built to integrate as seamlessly as possible with the [LangChain Python package](https://github.com/langchain-ai/langchain). Specifically, this means all objects (prompts, LLMs, chains, etc) are designed in a way where they can be serialized and shared between languages.
+## 🚀 How does LangChain help?
 
-The [LangChainHub](https://github.com/hwchase17/langchain-hub) is a central place for the serialized versions of these prompts, chains, and agents.
+The main value props of the LangChain libraries are:
+1. **Components**: composable tools and integrations for working with language models. Components are modular and easy-to-use, whether you are using the rest of the LangChain framework or not
+2. **Off-the-shelf chains**: built-in assemblages of components for accomplishing higher-level tasks
 
-## 🚀 What can this help with?
+Off-the-shelf chains make it easy to get started. Components make it easy to customize existing chains and build new ones. 
 
-There are six main areas that LangChain is designed to help with.
-These are, in increasing order of complexity:
+Components fall into the following **modules**:
 
-**📃 LLMs and Prompts:**
+**📃 Model I/O:**
 
 This includes prompt management, prompt optimization, a generic interface for all LLMs, and common utilities for working with LLMs.
 
-**🔗 Chains:**
-
-Chains go beyond a single LLM call and involve sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.
-
-**📚 Data Augmented Generation:**
+**📚 Retrieval:**
 
 Data Augmented Generation involves specific types of chains that first interact with an external data source to fetch data for use in the generation step. Examples include summarization of long pieces of text and question/answering over specific data sources.
 
 **🤖 Agents:**
 
 Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end-to-end agents.
 
-**🧠 Memory:**
-
-Memory refers to persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.
-
-**🧐 Evaluation:**
-
-[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.
-
-For more information on these concepts, please see our [full documentation](https://js.langchain.com/docs/).
+## 📖 Documentation
 
+Please see [here](https://js.langchain.com) for full documentation, which includes:
 
+- [Getting started](https://js.langchain.com/docs/get_started/introduction): installation, setting up the environment, simple examples
+- Overview of the [interfaces](https://js.langchain.com/docs/expression_language/), [modules](https://js.langchain.com/docs/modules/) and [integrations](https://js.langchain.com/docs/integrations/platforms)
+- [Use case](https://js.langchain.com/docs/use_cases/) walkthroughs and best practice [guides](https://js.langchain.com/docs/guides/)
+- [Reference](https://api.js.langchain.com): full API docs
 
 ## 💁 Contributing
 
-As an open source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infra, or better documentation.
+As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
 
 For detailed information on how to contribute, see [here](CONTRIBUTING.md).
 
 Please report any security issues or concerns following our [security guidelines](SECURITY.md).
+
+## 🖇️ Relationship with Python LangChain
+
+This is built to integrate as seamlessly as possible with the [LangChain Python package](https://github.com/langchain-ai/langchain). Specifically, this means all objects (prompts, LLMs, chains, etc) are designed in a way where they can be serialized and shared between languages.
+