Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: ChromaVectorStore can attempt to add in excess of chromadb batch… #8019

Merged
merged 5 commits into from
Oct 9, 2023
Merged

Fix: ChromaVectorStore can attempt to add in excess of chromadb batch… #8019

merged 5 commits into from
Oct 9, 2023

Conversation

Brad-Edwards
Copy link
Contributor

… size

Description

Changed ChromaVectorStore.add to batch inputs at one less than chromadb's max to avoid the oversize error. Adds a class method to do chunking.

Fixes #7648

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

I have passing unit tests on my branch but have not included them. The unit test to add a large number of nodes takes ~5 secs to run. If that's acceptable, I can submit the unit tests for add

I noticed tests were missing for all of the llama_index.vector_stores.chroma.py. So I also implemented tests for several other methods.

Let me know if you want all the unit tests, all the tests excluding add, or none of them.

  • [see above] Added new unit/integration tests
  • I stared at the code and made sure it makes sense

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • [N/A ] I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • [see above] I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

@logan-markewich logan-markewich merged commit ec7f434 into run-llama:main Oct 9, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Exceeding submit limit in ChromaVectorStore
3 participants