Merge pull request #282 from jakedetels/build-watch-mode-and-simpler-…

…examples-execution Improve contributor workflow
vercel · Feb 19, 2024 · 9cbb9b9 · 9cbb9b9
2 parents cabc62f + 80e6dca
commit 9cbb9b9
Show file tree

Hide file tree

Showing 39 changed files with 1,928 additions and 1,884 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -6,10 +6,10 @@ Feedback, bug reports and other contributions are welcome.
 
 > Pre-requisite: you have installed [git][install-git], [node][install-node] and [pnpm][install-pnpm].
 
-1. Clone the ModelFusion repository: `git clone https://github.com/lgrammel/modelfusion.git``
+1. Clone the ModelFusion repository: `git clone https://github.com/lgrammel/modelfusion.git`
 2. Go into the cloned repository: `cd modelfusion`
 3. Install dependencies: `pnpm install`
-4. Setup pre-commit hook: `pnpm run setup`. The precommit hook will format your changes ModelFusion each time you commit.
+4. Setup pre-commit hook: `pnpm run setup` (not `pnpm setup`). The precommit hook will format your changes ModelFusion each time you commit.
 5. Build: `pnpm build`
 
 ## Build
@@ -18,6 +18,34 @@ Feedback, bug reports and other contributions are welcome.
 pnpm build
 ```
 
+## Generate TypeScript declaration maps (.d.ts.map)
+
+ModelFusion is a monorepo consisting of many workspaces. When developing in the workspaces in `packages/*` and `tools/*`, you will want to enable TypeScript declaration maps (`.d.ts.map`). TypeScript declaration maps are mainly used to quickly jump to type definitions in the context of a monorepo (see [source issue](https://github.com/Microsoft/TypeScript/issues/14479) and [official documentation](https://www.typescriptlang.org/tsconfig/#declarationMap)). To enable TypeScript declaration maps, run the following:
+
+```sh
+pnpm build:dtsMap
+```
+
+With the TypeScript declaration maps created, you now can navigate to source `.ts` files more easily in your IDE. For example, consider the following:
+
+```ts
+// file: examples/basic/src/model-function/generate-text-example.ts
+import { generateText, openai } from "modelfusion";
+```
+
+In the above code snippet, you may be in VS Code and want to navigate to the source `.ts` file where `openai` is implemented. Without TypeScript declaration maps created, when you attempt to navigate to the source file for the imported `openai`, your IDE will open the compiled `.d.ts` file (`packages/modelfusion/dist/index.d.ts`). However, once you generate the TypeScript declaration maps, your IDE should now open the source `.ts` file (e.g., `packages/modelfusion/src/model-provider/openai/OpenAIFacade.ts`).
+
+## Watch Mode
+
+In addition to generating TypeScript declaration maps, you will want to recomplie your changes in real-time so that your changes are immediately available to the consuming packages in the monorepo. To do this, you can run the following:
+
+```sh
+cd packages/modelfusion # or whichever workspace you are developing in
+pnpm dev
+```
+
+The `pnpm dev` script will watch for changes in the specified workspace, recompile the changes, and produce updated TypeScript declaration maps.
+
 <!-- Links -->
 
 [install-git]: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

diff --git a/examples/babyagi-agent/README.md b/examples/babyagi-agent/README.md
@@ -12,16 +12,17 @@ TypeScript implementation of the classic [BabyAGI](https://github.com/yoheinakaj
 
 1. Create .env file with the following content:
 
-   ```
-   OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
-   ```
+```sh
+OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
+```
 
-2. Run the following commands:
+2. Run the following commands from the root directory of the modelfusion repo:
 
-   ```sh
-   pnpm install
-   pnpm tsx src/BabyAGI.ts --objective "Solve world hunger."
-   ```
+```sh
+pnpm install
+pnpm build
+pnpm tsx examples/babyagi-agent/src/BabyAGI.ts --objective "Solve world hunger."
+```
 
 ## BabyBeeAGI
 
@@ -31,14 +32,15 @@ TypeScript implementation of [BabyBeeAGI](https://github.com/yoheinakajima/babya
 
 1. Create .env file with the following content:
 
-   ```
-   OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
-   SERPAPI_API_KEY="YOUR_SERPAPI_API_KEY"
-   ```
+```sh
+OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
+SERPAPI_API_KEY="YOUR_SERPAPI_API_KEY"
+```
 
-2. Run the following commands:
+2. Run the following commands from the root directory of the modelfusion repo:
 
-   ```sh
-   pnpm install
-   pnpm tsx src/BabyBeeAGI.ts --objective "Research and write a short text about the BabyAGI project"
-   ```
+```sh
+pnpm install
+pnpm build
+pnpm tsx examples/babyagi-agent/src/BabyBeeAGI.ts --objective "Research and write a short text about the BabyAGI project"
+```
diff --git a/examples/babyagi-agent/package.json b/examples/babyagi-agent/package.json
@@ -12,7 +12,7 @@
     "commander": "10.0.1",
     "dotenv": "16.3.1",
     "jsdom": "^22.1.0",
-    "modelfusion": "0.136.0",
+    "modelfusion": "workspace:*",
     "serpapi": "^2.0.0"
   },
   "devDependencies": {

diff --git a/examples/basic/README.md b/examples/basic/README.md
@@ -6,20 +6,22 @@ Basic examples of how to the the functions in ModelFusion.
 
 1. Create .env file with the following content (and more settings, depending on the providers you want to use):
 
-   ```
-   OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
-   COHERE_API_KEY="YOUR_COHERE_API_KEY"
-   HUGGINGFACE_API_KEY="YOUR_HUGGINGFACE_API_KEY"
-   ...
-   ```
+```sh
+OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
+COHERE_API_KEY="YOUR_COHERE_API_KEY"
+HUGGINGFACE_API_KEY="YOUR_HUGGINGFACE_API_KEY"
+...
+```
 
-2. Setup:
+2. Run the following commands from the root directory of the modelfusion repo:
 
-   ```sh
-   pnpm install
-   ```
+```sh
+pnpm install
+pnpm build
+```
 
 3. Run any example:
-   ```sh
-   pnpm tsx src/path/to/example.ts
-   ```
+
+```sh
+pnpm tsx examples/basic/src/path/to/example.ts
+```
diff --git a/examples/basic/package.json b/examples/basic/package.json
@@ -4,16 +4,16 @@
   "license": "MIT",
   "private": true,
   "dependencies": {
-    "@modelfusion/google-custom-search-tool": "0.10.0",
-    "@modelfusion/mathjs-tool": "0.1.1",
-    "@modelfusion/pinecone": "0.4.0",
-    "@modelfusion/serpapi-tool": "0.11.0",
-    "@modelfusion/sqlite-vss": "0.4.0",
+    "@modelfusion/google-custom-search-tool": "workspace:*",
+    "@modelfusion/mathjs-tool": "workspace:*",
+    "@modelfusion/pinecone": "workspace:*",
+    "@modelfusion/serpapi-tool": "workspace:*",
+    "@modelfusion/sqlite-vss": "workspace:*",
+    "modelfusion": "workspace:*",
+    "modelfusion-experimental": "workspace:*",
     "@pinecone-database/pinecone": "0.1.6",
     "dotenv": "16.3.1",
     "secure-json-parse": "2.7.0",
-    "modelfusion": "0.136.0",
-    "modelfusion-experimental": "0.7.0",
     "nanoid": "3.3.6",
     "better-sqlite3": "9.0.0",
     "sqlite-vss": "0.1.2",

diff --git a/examples/basic/src/model-function/generate-transcription-example.ts b/examples/basic/src/model-function/generate-transcription-example.ts
@@ -1,14 +1,17 @@
 import dotenv from "dotenv";
 import { generateTranscription, openai } from "modelfusion";
 import fs from "node:fs";
+import path from "node:path";
 
 dotenv.config();
 
 async function main() {
   const transcription = await generateTranscription({
     model: openai.Transcriber({ model: "whisper-1" }),
     mimeType: "audio/mp3",
-    audioData: await fs.promises.readFile("data/test.mp3"),
+    audioData: await fs.promises.readFile(
+      path.join(__dirname, "../../data/test.mp3")
+    ),
   });
 
   console.log(transcription);

diff --git a/examples/basic/src/model-function/stream-text-multimodal-example.ts b/examples/basic/src/model-function/stream-text-multimodal-example.ts
@@ -6,7 +6,9 @@ import path from "node:path";
 dotenv.config();
 
 async function main() {
-  const image = fs.readFileSync(path.join("data", "example-image.png"));
+  const image = fs.readFileSync(
+    path.join(__dirname, "../../data/example-image.png")
+  );
 
   const textStream = await streamText({
     model: openai.ChatTextGenerator({

diff --git a/examples/basic/src/text-chunk/split-text-chunk-example.ts b/examples/basic/src/text-chunk/split-text-chunk-example.ts
@@ -1,12 +1,16 @@
 import dotenv from "dotenv";
 import { splitAtCharacter, splitTextChunk } from "modelfusion";
 import fs from "node:fs";
+import path from "node:path";
 
 dotenv.config();
 
 async function main() {
   const sanFranciscoWikipediaText = JSON.parse(
-    fs.readFileSync("data/san-francisco-wikipedia.json", "utf8")
+    fs.readFileSync(
+      path.join(__dirname, "../../data/san-francisco-wikipedia.json"),
+      "utf8"
+    )
   ).content as string;
 
   const chunks = await splitTextChunk(

diff --git a/examples/basic/src/text-chunk/split/split-at-character-example.ts b/examples/basic/src/text-chunk/split/split-at-character-example.ts
@@ -1,9 +1,13 @@
 import { splitAtCharacter } from "modelfusion";
 import fs from "node:fs";
+import path from "node:path";
 
 async function main() {
   const sanFranciscoWikipediaText = JSON.parse(
-    fs.readFileSync("data/san-francisco-wikipedia.json", "utf8")
+    fs.readFileSync(
+      path.join(__dirname, "../../../data/san-francisco-wikipedia.json"),
+      "utf8"
+    )
   ).content as string;
 
   const split = splitAtCharacter({ maxCharactersPerChunk: 1000 });

diff --git a/examples/basic/src/text-chunk/split/split-at-token-example.ts b/examples/basic/src/text-chunk/split/split-at-token-example.ts
@@ -1,9 +1,13 @@
 import { openai, splitAtToken } from "modelfusion";
 import fs from "node:fs";
+import path from "node:path";
 
 async function main() {
   const sanFranciscoWikipediaText = JSON.parse(
-    fs.readFileSync("data/san-francisco-wikipedia.json", "utf8")
+    fs.readFileSync(
+      path.join(__dirname, "../../../data/san-francisco-wikipedia.json"),
+      "utf8"
+    )
   ).content as string;
 
   const split = splitAtToken({

diff --git a/examples/basic/src/text-chunk/split/split-on-separator-example.ts b/examples/basic/src/text-chunk/split/split-on-separator-example.ts
@@ -1,9 +1,13 @@
 import { splitOnSeparator } from "modelfusion";
 import fs from "node:fs";
+import path from "node:path";
 
 async function main() {
   const sanFranciscoWikipediaText = JSON.parse(
-    fs.readFileSync("data/san-francisco-wikipedia.json", "utf8")
+    fs.readFileSync(
+      path.join(__dirname, "../../../data/san-francisco-wikipedia.json"),
+      "utf8"
+    )
   ).content as string;
 
   const split = splitOnSeparator({ separator: "\n" });

diff --git a/examples/chatbot-next-js/README.md b/examples/chatbot-next-js/README.md
@@ -8,15 +8,17 @@ A web chat with an AI assistant.
 
 1. Create `.env.local` file with the following content:
 
-```
+```sh
 OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
 ```
 
-2. Run the following commands:
+2. Run the following commands from the root directory of the modelfusion repo:
 
 ```sh
-npm install
-npm run dev
+pnpm install
+pnpm build
+cd examples/chatbot-next-js
+pnpm dev
 ```
 
 3. Go to http://localhost:3000/ in your browser
diff --git a/examples/chatbot-next-js/package.json b/examples/chatbot-next-js/package.json
@@ -20,8 +20,8 @@
     "@emotion/styled": "11.10.8",
     "@mui/icons-material": "5.15.5",
     "@mui/material": "5.12.2",
-    "modelfusion": "0.136.0",
-    "modelfusion-experimental": "0.7.0",
+    "modelfusion": "workspace:*",
+    "modelfusion-experimental": "workspace:*",
     "next": "13.5.1",
     "react": "18.2.0",
     "react-dom": "18.2.0",

diff --git a/examples/cloudflare-workers/README.md b/examples/cloudflare-workers/README.md
@@ -11,6 +11,14 @@ Generate text on a Cloudflare Worker using ModelFusion and OpenAI.
    - For local environment: Add to `.dev.vars`
    - For cloudflare workers: Use `pnpm wrangler secret put OPENAI_API_KEY`
 
-1. Run
+2. Run the following commands from the root directory of the modelfusion repo:
+
+```sh
+pnpm install
+pnpm build
+cd examples/cloudflare-workers
+```
+
+3. Run
    - Local environment: `pnpm dev`
    - Cloud (deployment): `pnpm deploy`
diff --git a/examples/cloudflare-workers/package.json b/examples/cloudflare-workers/package.json
@@ -15,6 +15,6 @@
     "wrangler": "^3.22.1"
   },
   "dependencies": {
-    "modelfusion": "0.136.0"
+    "modelfusion": "workspace:*"
   }
 }
diff --git a/examples/middle-school-math-agent/README.md b/examples/middle-school-math-agent/README.md
@@ -14,14 +14,10 @@ Note: GPT-4 can solve these problems without a calculator. This example is just
 OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
 ```
 
-2. Setup:
+2. Run the following commands from the root directory of the modelfusion repo:
 
 ```sh
 pnpm install
-```
-
-3. Run:
-
-```sh
-pnpm tsx src/MiddleSchoolMathAgent.ts
+pnpm build
+pnpm tsx examples/middle-school-math-agent/src/MiddleSchoolMathAgent.ts
 ```
diff --git a/examples/middle-school-math-agent/package.json b/examples/middle-school-math-agent/package.json
@@ -8,8 +8,8 @@
     "clean": "rimraf build dist .turbo node_modules && pnpm clear"
   },
   "dependencies": {
-    "@modelfusion/mathjs-tool": "0.1.1",
-    "modelfusion": "0.136.0",
+    "@modelfusion/mathjs-tool": "workspace:*",
+    "modelfusion": "workspace:*",
     "dotenv": "16.3.1",
     "zod": "3.22.4"
   },

diff --git a/examples/nextjs/README.md b/examples/nextjs/README.md
@@ -12,15 +12,17 @@ Examples of using ModelFusion with Next.js 14 (App Router):
 
 1. Create a `.env.local` file with the following content (depending on which demo you want to run)):
 
-```
+```sh
 OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
 STABILITY_API_KEY="YOUR_STABILITY_API_KEY"
 ```
 
-2. Run the following commands:
+2. Run the following commands from the root directory of the modelfusion repo:
 
 ```sh
 pnpm install
+pnpm build
+cd examples/nextjs
 pnpm dev
 ```
 

diff --git a/examples/nextjs/package.json b/examples/nextjs/package.json
@@ -15,8 +15,8 @@
     "class-variance-authority": "^0.7.0",
     "clsx": "^2.1.0",
     "lucide-react": "^0.312.0",
-    "modelfusion": "0.136.0",
-    "modelfusion-experimental": "0.7.0",
+    "modelfusion": "workspace:*",
+    "modelfusion-experimental": "workspace:*",
     "next": "14.1.0",
     "react": "^18",
     "react-dom": "^18",

diff --git a/examples/pdf-chat-terminal/README.md b/examples/pdf-chat-terminal/README.md
@@ -8,13 +8,14 @@ Ask questions about a PDF document and get answers from the document.
 
 1. Create .env file with the following content:
 
-```
+```sh
 OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
 ```
 
-2. Run the following commands:
+2. Run the following commands from the root directory of the modelfusion repo:
 
 ```sh
 pnpm install
-pnpm tsx src/main.ts -f my.pdf
+pnpm build
+pnpm tsx examples/pdf-chat-terminal/src/main.ts -f my.pdf
 ```