Nextjs invalid exports #824

jeffchuber · 2023-07-17T15:00:55Z

From discord: https://discord.com/channels/1073293645303795742/1130258136905814066/1130258136905814066

./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js
Module not found: Package path . is not exported from package C:\personal-dev\ai-prototypes-2\node_modules@visheratin\web-ai (see exports field in C:\personal-dev\ai-prototypes-2\node_modules@visheratin\web-ai\package.json)

Import trace for requested module:
./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js

Anyone know how i can make webpack or nextjs not through these module not found issues? It seems like it shouldn't be importing this unless the function is actually being ran?

The text was updated successfully, but these errors were encountered:

dosu-bot · 2023-07-17T16:13:47Z

Answer generated by a 🤖

Answer

From your description, it appears that you're encountering an issue with Next.js where it's unable to find the module @visheratin/web-ai. This is likely due to the package not correctly exporting its main file, which is a common problem when using ES6 imports with CommonJS modules.

To address this, you could modify the package.json file of the @visheratin/web-ai package to include an exports field. This field should specify the main file to be exported when the package is imported. Here's an example of how you could set it up:

{
  "name": "@visheratin/web-ai",
  "version": "1.0.0",
  "main": "index.js",
  "exports": {
    ".": "./index.js"
  }
}

In this example, the exports field is indicating that the main file (index.js) should be exported when the package is imported.

However, if you don't have the ability to modify the package.json of the @visheratin/web-ai package, you could try using a dynamic import() statement instead of a static require() statement. This can sometimes bypass issues with ES6 imports and CommonJS modules.

Here's an example of how you could adjust the WebAIEmbeddingFunction class to use a dynamic import() statement:

export class WebAIEmbeddingFunction implements IEmbeddingFunction {
  // ...
  constructor(
    modality: "text" | "image",
    node: boolean,
    proxy?: boolean,
    wasmPath?: string,
    modelID?: string
  ) {
    if (node) {
      this.proxy = proxy ? proxy : false;
      import("@visheratin/web-ai-node")
        .then((module) => {
          webAI = module;
        })
        .catch((e) => {
          console.log(e);
          throw new Error(
            "Please install the @visheratin/web-ai-node package to use the WebAIEmbeddingFunction, `npm install -S @visheratin/web-ai-node`"
          );
        });
    } else {
      this.proxy = proxy ? proxy : true;
      import("@visheratin/web-ai")
        .then((module) => {
          webAI = module;
        })
        .catch((e) => {
          console.log(e);
          throw new Error(
            "Please install the @visheratin/web-ai package to use the WebAIEmbeddingFunction, `npm install -S @visheratin/web-ai`"
          );
        });
    }
    // ...
  }
  // ...
}

In this revised version of the WebAIEmbeddingFunction class, the require() statements have been replaced with import() statements. These import() statements return a promise that resolves to the imported module, which is then assigned to the webAI variable.

Please note that this is just one potential solution to your problem. If you find this solution to be correct and believe it could benefit other users, I encourage you to make a pull request. If this solution doesn't align with your experience, I recommend updating to the latest version of the software.

Sources

clients/js/src/embeddings/WebAIEmbeddingFunction.ts

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

e-roy · 2023-07-18T16:54:06Z

I got this error with using Next JS using the newer app router

I rolled back to "chromadb": "1.5.2", and no longer saw the error.

cishiv · 2023-07-19T08:53:50Z

I got this error with using Next JS using the newer app router

I rolled back to "chromadb": "1.5.2", and no longer saw the error.

Chiming in here - confirmed that rolling back to 1.5.2 has worked around the error on my end.

jeffchuber · 2023-07-21T05:33:35Z

@cishiv thanks!! if anyone can help me with a small reproducible example - id love to fix this

e-roy · 2023-07-23T11:44:22Z

@cishiv thanks!! if anyone can help me with a small reproducible example - id love to fix this

Hi @jeffchuber

Here is a repo:
https://github.com/e-roy/langchain-chatbot-demo

This is currently using "chromadb": "1.5.2"

If I upgrade to 1.5.5, it will still work, but there will be Module not found errors in the terminal

If I go through the process of adding each dependency @visheratin/web-ai-node, @visheratin/web-ai and encoding, then I'll get the error from the original post.

I hope this helps

jeffchuber · 2023-07-23T22:21:01Z

thank you @e-roy ! asking Vercel for help on this

shuding · 2023-07-25T12:39:58Z

From a quick look I think the problem is that the CJS build of chromadb was used (dist/main here instead of dist/module):

./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js

And then, a require is called:

require("@visheratin/web-ai")

So it tries to load the CJS build of @visheratin/web-ai. However, in that package's exports field, it has only types and import conditions for the . path defined: https://unpkg.com/browse/@visheratin/[email protected]/package.json

"exports": {
  ".": {
    "types": "./index.d.ts",
    "import": "./index.js"
  },

Which doesn't have a CJS build provided for that path at all. To fix, you have to either use import() to load the ESM build, or add a "main" field to its package.json, or add a "require" condition (https://nodejs.org/api/packages.html#conditional-exports).

visheratin · 2023-08-04T03:53:31Z

@shuding, thank you for the lead! I just opened a PR that addresses the issue with require/import statements. I tested the fixed version, and it works in both browser and Node.js.

## Description of changes After the recent changes in the Web AI structure, the embedder function became invalid because of different imports and because the browser version of the library is only ESM, as mentioned [here](#824 (comment)). To fix the issue, I updated the import paths and changed the `require` statements to `import` for the browser version of the function. **Important note:** When using the embedded in the browser, you should import `chromadb/dist/module` because the [browser version](https://www.npmjs.com/package/@visheratin/web-ai) of Web AI is ESM-only. For Node.js, you can use regular `chromadb` import because [Node.js version](https://www.npmjs.com/package/@visheratin/web-ai-node) of the library is CJS. Additionally, because Web AI since recently supports multimodal embeddings, I added building multimodal embeddings using CLIP-base. ## Test plan I tested the changes locally for both browser and Node.js. The built `chromadb` package if someone wants to test it in their projects, is available [here](https://drive.google.com/file/d/1cNLsHGd1VmiFiamvsEaMVGA7ng56QQKG/view?usp=sharing). ## Documentation Changes There likely will be needed changes regarding the multimodal functionality.

perzeuss · 2023-08-10T17:51:18Z

From a quick look I think the problem is that the CJS build of chromadb was used (dist/main here instead of dist/module):
./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js
And then, a require is called:
require("@visheratin/web-ai")
So it tries to load the CJS build of @visheratin/web-ai. However, in that package's exports field, it has only types and import conditions for the . path defined: https://unpkg.com/browse/@visheratin/[email protected]/package.json
"exports": {
  ".": {
    "types": "./index.d.ts",
    "import": "./index.js"
  },
Which doesn't have a CJS build provided for that path at all. To fix, you have to either use import() to load the ESM build, or add a "main" field to its package.json, or add a "require" condition (https://nodejs.org/api/packages.html#conditional-exports).

You will still have this error when you use the ESM build.
See

- wait compiling /_error (client and server)...
- error ../chroma/clients/js/dist/module/embeddings/WebAIEmbeddingFunction.js:132:0
Module not found: Can't resolve '@visheratin/web-ai'
  130 |         try {
  131 |             // @ts-ignore
> 132 |             const webAI = await import("@visheratin/web-ai");
  133 |             if (wasmPath) {
  134 |                 webAI.SessionParams.wasmRoot = wasmPath;
  135 |             }

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
../chroma/clients/js/dist/module/index.js
./src/app/page.js

The problem here is, that the package @visheratin/web-ai is not installed which is intended because this module is optional and there is a try-catch around that import to catch that error at runtime. However, the Next.js build script tries to locate the module during the compile process and throws this error because it is not installed.

@shuding can you confirm that?

visheratin · 2023-08-10T19:13:57Z

I just checked and can confirm that this issue exists. When working on a fix, I didn't consider this use case.

To fix the issue in version 1.5.6 of the package for Next.js, you can add the following to the Webpack config in next.config.js:

config.resolve.fallback = {
      "@visheratin/web-ai": false
    };

This way, the app will get compiled.

If somebody knows how to fix this issue in the WebAIEmbeddingFunction, I'd greatly appreciate the help.

perzeuss · 2023-08-10T19:28:02Z

I just checked and can confirm that this issue exists. When working on a fix, I didn't consider this use case.

To fix the issue in version 1.5.6 of the package for Next.js, you can add the following to the Webpack config in next.config.js:
config.resolve.fallback = {
      "@visheratin/web-ai": false
    };
This way, the app will get compiled.

If somebody knows how to fix this issue in the WebAIEmbeddingFunction, I'd greatly appreciate the help.

I already fixed the WebAIEmbeddingFunction in #956

GallagherSam · 2023-08-25T02:23:47Z

I was having this same issue on a Vue app I'm working on. I found the following script roughly fixes the import problem so my app will build properly. Not a great solution, but works.

input_file="node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js"
output_file="node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js"

while IFS= read -r line; do
    if [[ $line == *'require("@visheratin/'* ]]; then
        line="//${line}"
    fi
    echo "$line"
done < "$input_file" > "$output_file"

jeffchuber · 2023-08-27T22:30:02Z

we are close to fixing this up thanks to some excellent work by @perzeuss ! stand by :)

l4b4r4b4b4 · 2023-08-30T12:05:09Z

well unfortunately neither the PR was merged nor does the custom webpack config work for me. Would be great not to have to build a workaround or rolling back to prev. versions to fix the issue.

jeffchuber · 2023-09-06T03:28:06Z

This should be fixed in the release 1.5.8 - can folks give that a spin?

jonsoku-dev · 2023-09-06T05:52:01Z

@jeffchuber
The error is continuously displayed in version 1.5.8 when ingest documents.

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

-  ┌ POST /api/chat/retrieval 200 in 21499ms
   │
   ├──── POST http://127.0.0.1:7080/api/v1/collections 200 in 32ms (cache: MISS)
   │
   ├── 1 level ── POST http://127.0.0.1:11434/api/generate 200 in 3215ms (cache: MISS)
   │
   └── 1 level ── POST http://127.0.0.1:7080/api/v1/collections/1038.. 200 in 66ms (cache: MISS)

perzeuss · 2023-09-07T16:25:56Z

@jeffchuber The error is continuously displayed in version 1.5.8 when ingest documents.

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

-  ┌ POST /api/chat/retrieval 200 in 21499ms
   │
   ├──── POST http://127.0.0.1:7080/api/v1/collections 200 in 32ms (cache: MISS)
   │
   ├── 1 level ── POST http://127.0.0.1:11434/api/generate 200 in 3215ms (cache: MISS)
   │
   └── 1 level ── POST http://127.0.0.1:7080/api/v1/collections/1038.. 200 in 66ms (cache: MISS)

@jonsoku2 Are you trying to use the webai embedding function or only the chroma client?

e-roy · 2023-09-16T14:08:38Z

This seems to be working fine now with Next JS, using:
"chromadb": "^1.5.9",
"langchain": "^0.0.150",
"next": "^13.4.19",

Here is a working example:

https://github.com/e-roy/langchain-chatbot-demo/tree/main

For this error:

Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts

You can adjust your next.config.js file and ignore it by adding

  webpack: (config, {}) => {
    config.resolve.fallback = {
      "@visheratin/web-ai-node": false,
    };
    return config;
  },

l4b4r4b4b4 · 2023-09-16T14:31:36Z

This seems to be working fine now with Next JS, using: "chromadb": "^1.5.9", "langchain": "^0.0.150", "next": "^13.4.19",

Here is a working example:

https://github.com/e-roy/langchain-chatbot-demo/tree/main

For this error:
Module not found: Can't resolve '@visheratin/web-ai-node/multimodal' in '/Users/xxx/Documents/GitHub/langchain/javascript/langchain-nextjs/node_modules/chromadb/dist/module/embeddings'

Import trace for requested module:
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js
./node_modules/chromadb/dist/module/index.js
./node_modules/langchain/dist/vectorstores/chroma.js
./node_modules/langchain/vectorstores/chroma.js
./app/api/retrieval/ingest/route.ts
You can adjust your next.config.js file and ignore it by adding
  webpack: (config, {}) => {
    config.resolve.fallback = {
      "@visheratin/web-ai-node": false,
    };
    return config;
  },

interesting! Will try out. Thx for notifying!

jeffchuber · 2023-11-07T19:23:24Z

This is fixed with 1.6.1

visheratin mentioned this issue Aug 4, 2023

Web AI embedder fixes #929

Merged

jeffchuber mentioned this issue Aug 10, 2023

fix: use workaround for dynamic import #956

Merged

jeffchuber closed this as completed Nov 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nextjs invalid exports #824

Nextjs invalid exports #824

jeffchuber commented Jul 17, 2023

dosu-bot commented Jul 17, 2023

e-roy commented Jul 18, 2023

cishiv commented Jul 19, 2023

jeffchuber commented Jul 21, 2023

e-roy commented Jul 23, 2023

jeffchuber commented Jul 23, 2023

shuding commented Jul 25, 2023

visheratin commented Aug 4, 2023

perzeuss commented Aug 10, 2023 •

edited

Loading

visheratin commented Aug 10, 2023

perzeuss commented Aug 10, 2023

GallagherSam commented Aug 25, 2023

jeffchuber commented Aug 27, 2023

l4b4r4b4b4 commented Aug 30, 2023

jeffchuber commented Sep 6, 2023

jonsoku-dev commented Sep 6, 2023 •

edited

Loading

perzeuss commented Sep 7, 2023 •

edited

Loading

e-roy commented Sep 16, 2023 •

edited

Loading

l4b4r4b4b4 commented Sep 16, 2023

jeffchuber commented Nov 7, 2023

Nextjs invalid exports #824

Nextjs invalid exports #824

Comments

jeffchuber commented Jul 17, 2023

dosu-bot commented Jul 17, 2023

Answer

Sources

e-roy commented Jul 18, 2023

cishiv commented Jul 19, 2023

jeffchuber commented Jul 21, 2023

e-roy commented Jul 23, 2023

jeffchuber commented Jul 23, 2023

shuding commented Jul 25, 2023

visheratin commented Aug 4, 2023

perzeuss commented Aug 10, 2023 • edited Loading

visheratin commented Aug 10, 2023

perzeuss commented Aug 10, 2023

GallagherSam commented Aug 25, 2023

jeffchuber commented Aug 27, 2023

l4b4r4b4b4 commented Aug 30, 2023

jeffchuber commented Sep 6, 2023

jonsoku-dev commented Sep 6, 2023 • edited Loading

perzeuss commented Sep 7, 2023 • edited Loading

e-roy commented Sep 16, 2023 • edited Loading

l4b4r4b4b4 commented Sep 16, 2023

jeffchuber commented Nov 7, 2023

perzeuss commented Aug 10, 2023 •

edited

Loading

jonsoku-dev commented Sep 6, 2023 •

edited

Loading

perzeuss commented Sep 7, 2023 •

edited

Loading

e-roy commented Sep 16, 2023 •

edited

Loading