Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: create downstream vectordb (pinecone) connector #1414

Closed
rbiseck3 opened this issue Sep 14, 2023 · 0 comments · Fixed by #1774
Closed

feat: create downstream vectordb (pinecone) connector #1414

rbiseck3 opened this issue Sep 14, 2023 · 0 comments · Fixed by #1774
Assignees
@ahmetmeleq

Comments

@rbiseck3
Copy link
Contributor

Create downstream connector for Pinecone vector database. This will depend on #1319 for full completeness as embeddings are required to add the content into a vector db. Does not seem that Pinecone has a supported docker approach for local testing so will have to use a Pinecone instance with an associated api key to test against.
@rbiseck3 rbiseck3 self-assigned this Sep 14, 2023
@rbiseck3 rbiseck3 converted this from a draft issue Sep 14, 2023
@rbiseck3 rbiseck3 moved this from Blocked to Todo in unstructured-ingest Sep 21, 2023
@ahmetmeleq ahmetmeleq mentioned this issue Oct 17, 2023
Merged
@ryannikolaidis ryannikolaidis moved this from Todo to In Progress in unstructured-ingest Oct 17, 2023
github-merge-queue bot pushed a commit that referenced this issue Nov 29, 2023
@ahmetmeleq @rbiseck3
feat: add pinecone destination connector (#1774)
ed08773 
Closes #1414
Closes #2039 

This PR:
- Uses Pinecone python cli to implement a destination connector for
Pinecone and provides the ingest readme requirements
[(here)](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest#the-checklist)
for the connector
- Updates documentation for the s3 destination connector
- Alphabetically sorts setup.py contents
- Updates logs for the chunking node  in ingest pipeline
- Adds a baseline session handle implementation for destination
connectors, to be able to parallelize their operations
- For the
[bug](#1892)
related to persisting element data to ingest embedding nodes; this PR
tests the
[solution](#1893)
with its ingest test
- Solves a bug on ingest chunking params with [bugfix on chunking params
and implementing related
test](69e1949)

---------

Co-authored-by: Roman Isecke <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Done in unstructured-ingest Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahmetmeleq ahmetmeleq

Labels
None yet
Projects
unstructured-ingest
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: add pinecone destination connector Unstructured-IO/unstructured
2 participants
@ahmetmeleq @rbiseck3