Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add ObsidianLoader to Document loaders #3494

Merged
merged 6 commits into from
Dec 5, 2023
Merged

feat: Add ObsidianLoader to Document loaders #3494

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
32840b3
Add ObsidianLoader integration
eactisgrosso Dec 2, 2023
25d408d
Merge remote-tracking branch 'upstream/main'
eactisgrosso Dec 3, 2023
ca6dfdb
Fix Notion test not to consider Obsidian '.md' files
eactisgrosso Dec 3, 2023
a8b5224
Merge branch 'main' of https://github.com/hwchase17/langchainjs into …
jacoblee93 Dec 4, 2023
912027e
Fix lint
jacoblee93 Dec 4, 2023
9472b65
Merge branch 'main' of https://github.com/hwchase17/langchainjs into …
jacoblee93 Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api_refs/typedoc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@
"./langchain/src/document_loaders/fs/epub.ts",
"./langchain/src/document_loaders/fs/csv.ts",
"./langchain/src/document_loaders/fs/notion.ts",
"./langchain/src/document_loaders/fs/obsidian.ts",
"./langchain/src/document_loaders/fs/unstructured.ts",
"./langchain/src/document_loaders/fs/openai_whisper_audio.ts",
"./langchain/src/document_loaders/fs/pptx.ts",
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-bun/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export * from "langchain/document_loaders/base";
export * from "langchain/document_loaders/web/searchapi";
export * from "langchain/document_loaders/web/serpapi";
export * from "langchain/document_loaders/web/sort_xyz_blockchain";
export * from "langchain/document_loaders/fs/obsidian";
export * from "langchain/document_transformers/openai_functions";
export * from "langchain/chat_models/base";
export * from "langchain/chat_models/openai";
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-cf/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export * from "langchain/document_loaders/base";
export * from "langchain/document_loaders/web/searchapi";
export * from "langchain/document_loaders/web/serpapi";
export * from "langchain/document_loaders/web/sort_xyz_blockchain";
export * from "langchain/document_loaders/fs/obsidian";
export * from "langchain/document_transformers/openai_functions";
export * from "langchain/chat_models/base";
export * from "langchain/chat_models/openai";
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-cjs/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ const document_loaders_base = require("langchain/document_loaders/base");
const document_loaders_web_searchapi = require("langchain/document_loaders/web/searchapi");
const document_loaders_web_serpapi = require("langchain/document_loaders/web/serpapi");
const document_loaders_web_sort_xyz_blockchain = require("langchain/document_loaders/web/sort_xyz_blockchain");
const document_loaders_fs_obsidian = require("langchain/document_loaders/fs/obsidian");
const document_transformers_openai_functions = require("langchain/document_transformers/openai_functions");
const chat_models_base = require("langchain/chat_models/base");
const chat_models_openai = require("langchain/chat_models/openai");
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-esbuild/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import * as document_loaders_base from "langchain/document_loaders/base";
import * as document_loaders_web_searchapi from "langchain/document_loaders/web/searchapi";
import * as document_loaders_web_serpapi from "langchain/document_loaders/web/serpapi";
import * as document_loaders_web_sort_xyz_blockchain from "langchain/document_loaders/web/sort_xyz_blockchain";
import * as document_loaders_fs_obsidian from "langchain/document_loaders/fs/obsidian";
import * as document_transformers_openai_functions from "langchain/document_transformers/openai_functions";
import * as chat_models_base from "langchain/chat_models/base";
import * as chat_models_openai from "langchain/chat_models/openai";
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-esm/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import * as document_loaders_base from "langchain/document_loaders/base";
import * as document_loaders_web_searchapi from "langchain/document_loaders/web/searchapi";
import * as document_loaders_web_serpapi from "langchain/document_loaders/web/serpapi";
import * as document_loaders_web_sort_xyz_blockchain from "langchain/document_loaders/web/sort_xyz_blockchain";
import * as document_loaders_fs_obsidian from "langchain/document_loaders/fs/obsidian";
import * as document_transformers_openai_functions from "langchain/document_transformers/openai_functions";
import * as chat_models_base from "langchain/chat_models/base";
import * as chat_models_openai from "langchain/chat_models/openai";
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-vercel/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export * from "langchain/document_loaders/base";
export * from "langchain/document_loaders/web/searchapi";
export * from "langchain/document_loaders/web/serpapi";
export * from "langchain/document_loaders/web/sort_xyz_blockchain";
export * from "langchain/document_loaders/fs/obsidian";
export * from "langchain/document_transformers/openai_functions";
export * from "langchain/chat_models/base";
export * from "langchain/chat_models/openai";
Expand Down
1 change: 1 addition & 0 deletions environment_tests/test-exports-vite/src/entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ export * from "langchain/document_loaders/base";
export * from "langchain/document_loaders/web/searchapi";
export * from "langchain/document_loaders/web/serpapi";
export * from "langchain/document_loaders/web/sort_xyz_blockchain";
export * from "langchain/document_loaders/fs/obsidian";
export * from "langchain/document_transformers/openai_functions";
export * from "langchain/chat_models/base";
export * from "langchain/chat_models/openai";
Expand Down
9 changes: 9 additions & 0 deletions examples/src/document_loaders/example_data/obsidian/bad_frontmatter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
anArray:
one
- two
- three
tags: 'onetag', 'twotag' ]
---

A document with frontmatter that isn't valid.
5 changes: 5 additions & 0 deletions examples/src/document_loaders/example_data/obsidian/frontmatter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
tags: journal/entry, obsidian
---

No other content than the frontmatter.
13 changes: 13 additions & 0 deletions examples/src/document_loaders/example_data/obsidian/no_frontmatter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
### Description
#recipes #dessert #cookies

A document with HR elements that might trip up a front matter parser:

---

### Ingredients

- 3/4 cup (170g) **unsalted butter**, slightly softened to room temperature.
- 1 and 1/2 cups (180g) **confectioners’ sugar**

---
1 change: 1 addition & 0 deletions examples/src/document_loaders/example_data/obsidian/no_metadata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A markdown document with no additional metadata.
35 changes: 35 additions & 0 deletions examples/src/document_loaders/example_data/obsidian/tags_and_frontmatter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
aFloat: 13.12345
anInt: 15
aBool: true
aString: string value
anArray:
- one
- two
- three
aDict:
dictId1: '58417'
dictId2: 1500
tags: [ 'onetag', 'twotag' ]
---

# Tags

()#notatag
#12345
#read
something #tagWithCases
- #tag-with-dash
#tag_with_underscore #tag/with/nesting

# Dataview

Here is some data in a [dataview1:: a value] line.
Here is even more data in a (dataview2:: another value) line.
dataview3:: more data
notdataview4: this is not a field
notdataview5: this is not a field

# Text content

https://example.com/blog/#not-a-tag
9 changes: 9 additions & 0 deletions examples/src/document_loaders/obsidian.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { ObsidianLoader } from "langchain/document_loaders/fs/obsidian";

export const run = async () => {
const loader = new ObsidianLoader("src/document_loaders/example_data/obsidian");

const docs = await loader.load();

console.log({ docs });
};
3 changes: 3 additions & 0 deletions langchain/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,9 @@ document_loaders/fs/csv.d.ts
document_loaders/fs/notion.cjs
document_loaders/fs/notion.js
document_loaders/fs/notion.d.ts
document_loaders/fs/obsidian.cjs
document_loaders/fs/obsidian.js
document_loaders/fs/obsidian.d.ts
document_loaders/fs/unstructured.cjs
document_loaders/fs/unstructured.js
document_loaders/fs/unstructured.d.ts
Expand Down
8 changes: 8 additions & 0 deletions langchain/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,6 +478,9 @@
"document_loaders/fs/notion.cjs",
"document_loaders/fs/notion.js",
"document_loaders/fs/notion.d.ts",
"document_loaders/fs/obsidian.cjs",
"document_loaders/fs/obsidian.js",
"document_loaders/fs/obsidian.d.ts",
"document_loaders/fs/unstructured.cjs",
"document_loaders/fs/unstructured.js",
"document_loaders/fs/unstructured.d.ts",
Expand Down Expand Up @@ -2233,6 +2236,11 @@
"import": "./document_loaders/fs/notion.js",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there! 👋 I noticed that this PR introduces a new dependency related to document loading for Obsidian. This change may impact the project's dependencies, so I'm flagging it for the maintainers to review. Great work on the PR! 🚀

"require": "./document_loaders/fs/notion.cjs"
},
"./document_loaders/fs/obsidian": {
"types": "./document_loaders/fs/obsidian.d.ts",
"import": "./document_loaders/fs/obsidian.js",
"require": "./document_loaders/fs/obsidian.cjs"
},
"./document_loaders/fs/unstructured": {
"types": "./document_loaders/fs/unstructured.d.ts",
"import": "./document_loaders/fs/unstructured.js",
Expand Down
1 change: 1 addition & 0 deletions langchain/scripts/create-entrypoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@ const entrypoints = {
"document_loaders/fs/epub": "document_loaders/fs/epub",
"document_loaders/fs/csv": "document_loaders/fs/csv",
"document_loaders/fs/notion": "document_loaders/fs/notion",
"document_loaders/fs/obsidian": "document_loaders/fs/obsidian",
"document_loaders/fs/unstructured": "document_loaders/fs/unstructured",
"document_loaders/fs/openai_whisper_audio":
"document_loaders/fs/openai_whisper_audio",
Expand Down
Loading