Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ingesting Confluence. #57

Open
homanp opened this issue Feb 22, 2024 · 0 comments
Open

Add support for ingesting Confluence. #57

homanp opened this issue Feb 22, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@homanp
Copy link
Contributor

homanp commented Feb 22, 2024

Describe the issue

A user should be able to ingest documents from confluence by passing the following payload to the /ingest API endpoint:

{
  "index_name": "string",
  "vector_database": {
    "type": "qdrant",
    "config": {}
  },
  "encoder": {
      "dimensions": 384,
      "model_name": "embed-multilingual-light-v3.0",
      "provider": "cohere"
    },
  "confluence": {
    "api_token": <api_token>,
    "user_email": < user_email >,
    "url":  <url>,
  },
  "webhook_url": "string"
}

Ingestion

After load process has finished we need to loop over the downloaded documents and use the embedding service to chunk, split, generate and save embeddings to the embedding db. The Embedding Service and walkthroughs should be sufficient for this.

Relevant files

Route ingestion on request: https://github.com/superagent-ai/super-rag/blob/main/api/ingest.py#L23
Create a new handler: https://github.com/superagent-ai/super-rag/blob/main/service/ingest.py

@homanp homanp added the enhancement New feature or request label Feb 22, 2024
@homanp homanp self-assigned this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant