Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find optimal embedding text chunk size/strategy #4

Open
mbertrand opened this issue Jul 25, 2023 · 0 comments
Open

Find optimal embedding text chunk size/strategy #4

mbertrand opened this issue Jul 25, 2023 · 0 comments
Assignees
@mbertrand

Comments

@mbertrand
Copy link
Member

mbertrand commented Jul 25, 2023 •
edited
Loading

Experiment with different text chunking strategies, start by simply increasing the # of characters to 512 instead of 2xx for a few courses to see if results improve compared to the current length. Also look into more complex strategies similar to that described here. The langchain package contains a RecursiveCharacterTextSplitter that could also be beneficial for this.
@mbertrand mbertrand self-assigned this Jul 25, 2023
@mbertrand mbertrand changed the title Find optimal embedding text chunk size Find optimal embedding text chunk size/strategy Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mbertrand mbertrand

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mbertrand