(EAI-18): Docs for RAG framework (#243)

* start stubbing out the docs

* docs v1

* fix build issues

* add data sources page

* copy edits

* update chat-ui docs

* Add back demo gif

chat-server/README.md

@@ -260,7 +260,3 @@ To format the code, run:
 npm run format
 
 ```
-
-```
-
-```

docs/.gitignore

@@ -0,0 +1,20 @@
+# Dependencies
+/node_modules
+
+# Production
+/build
+
+# Generated files
+.docusaurus
+.cache-loader
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

docs/README.md

@@ -0,0 +1,41 @@
+# Website
+
+This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
+
+### Installation
+
+```
+$ yarn
+```
+
+### Local Development
+
+```
+$ yarn start
+```
+
+This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
+
+### Build
+
+```
+$ yarn build
+```
+
+This command generates static content into the `build` directory and can be served using any static contents hosting service.
+
+### Deployment
+
+Using SSH:
+
+```
+$ USE_SSH=true yarn deploy
+```
+
+Not using SSH:
+
+```
+$ GIT_USER=<Your GitHub username> yarn deploy
+```
+
+If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.

docs/babel.config.js

@@ -0,0 +1,3 @@
+module.exports = {
+  presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
+};

docs/docs/index.md

@@ -0,0 +1,48 @@
+# MongoDB RAG Framework
+
+Build full stack retrieval augmented generation (RAG) applications using MongoDB
+and [Atlas Vector Search](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/).
+
+This framework is used to build the MongoDB Docs Chatbot, a RAG chatbot that answers questions about the MongoDB documentation. You can try it out on [mongodb.com/docs](https://www.mongodb.com/docs/).
+
+## How It Works
+
+The MongoDB RAG Framework has the following core components:
+
+- [MongoDB Atlas](./mongodb.md): Database for the application that stores content and conversation.
+  Indexes content using Atlas Vector Search.
+- [Ingest CLI](./ingest/configure.md): Configurable CLI application that you can use to ingest content into a MongoDB collection for use with Atlas Vector Search.
+- [Chat Server](./server/configure.md): Express.js server routes that you can use to build a chatbot application.
+- [Chat UI](./ui.md): React.js UI components that you can use to build a chatbot application.
+
+## Quick Start
+
+To get started using the MongoDB RAG framework, refer to the [Quick Start](./quick-start.md) guide.
+
+## Design Principles
+
+The MongoDB RAG Framework is designed around the following principles:
+
+- Composability: You can use components of the RAG framework independently of each other.
+  For example, we have some users who are using only our ingestion CLI to ingest content into MongoDB Atlas, but use other tools to build their chatbot and UI.
+- Pluggability: You can plug in your own implementations of components.
+  For example, you can plug in your own implementations of the `DataSource` interface
+  to ingest content from different data sources.
+- Inversion of Control: The RAG framework makes decisions about boilerplate aspects
+  of RAG systems so that you can focus on building logic unique to your application.
+
+## Architecture
+
+Here's a reference architecture for how the MongoDB RAG system works for the MongoDB Docs Chatbot.
+
+Data ingestion:
+
+![Data Ingestion Architecture](/img/ingest-diagram.webp)
+
+Chat Server:
+
+![Chat Server Architecture](/img/server-diagram.webp)
+
+## How We Built It
+
+- To learn more about how we built the chatbot, check out the MongoDB Developer Center blog post [Taking RAG to Production with the MongoDB Documentation AI Chatbot](https://www.mongodb.com/developer/products/atlas/taking-rag-to-production-documentation-ai-chatbot/).

docs/docs/ingest/command-reference.md

@@ -0,0 +1,47 @@
+# CLI Commands Reference
+
+The MongoDB RAG Ingest CLI has the following commands.
+
+You must first install the CLI before you can run it. See the [installation instructions in the Configuration documentation](./configure.md#install-the-ingest-cli).
+
+For all commands, you can use the `--help` flag to get more information about the command.
+
+## `pages`
+
+Update `pages` data from [data sources](data-sources.md).
+
+Options:
+
+```txt
+  --version  Show version number                                       [boolean]
+  --help     Show help                                                 [boolean]
+  --config   Path to config JS file.                                    [string]
+  --source   A source name to load. If unspecified, loads all sources.  [string]
+```
+
+## `embed`
+
+Update `embedded_content` data from the `pages` data.
+
+Options:
+
+```txt
+  --version  Show version number                                       [boolean]
+  --help     Show help                                                 [boolean]
+  --config   Path to config JS file.                                    [string]
+  --source   A source name to load. If unspecified, loads all sources.  [string]
+  --since                                                    [string] [required]
+```
+
+## `all`
+
+Run 'pages' and 'embed' for all data sources since last successful run of `all` command.
+On the first run, it will run for all data sources.
+
+Options:
+
+```txt
+  --version  Show version number                                       [boolean]
+  --help     Show help                                                 [boolean]
+  --config   Path to config JS file.                                    [string]
+```