Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Knowledge - Errors seen when ingesting lot of knowledge files. #65

Closed
sangee2004 opened this issue Sep 25, 2024 · 9 comments
Closed

Knowledge - Errors seen when ingesting lot of knowledge files. #65

sangee2004 opened this issue Sep 25, 2024 · 9 comments
Labels
bug Something isn't working knowledge

Comments

@sangee2004
Copy link

Server - f2fecd35e6

UI - f021415137

Steps to reproduce the problem:

  1. Create an agent and add lot of knowledge files (~500 in my case)
  2. Ingestion of files start and then UI quits after some time .
  3. I see a lot id these failures in the server logs repeatedly.
ERRO[0418] failed to update file: Operation cannot be fulfilled on KnowledgeFile.otto.gptscript.ai "e37d9f01a5c006a7f64bfb24f4898faeede81b4bcab2cefdc7db40e328fa1459": object has been modified, please apply your changes to the latest version and try again  logger=/pkg/controller/handlers/knowledge
ERRO[0418] sql query error                               affected=0 caller="/Users/sangeethahariharan/go/pkg/mod/github.com/acorn-io/[email protected]/pkg/db/db.go:686" elapsed="24.25µs" error="UNIQUE constraint failed: knowledgefile.previous" sql="INSERT INTO `knowledgefile` (`kind`,`version`,`api_group`,`name`,`namespace`,`uid`,`generation`,`previous`,`create`,`created`,`updated`,`deleted`,`removed`,`garbage`,`latest`,`metadata`,`data`,`status`,`partition_id`) VALUES (\"KnowledgeFile\",\"v1\",\"otto.gptscript.ai\",\"238c1115fa9e41cf31f1d0b7470fb5892696fa6f7679735fa65a51a74845f09b\",\"default\",\"89d35470-3790-4330-9754-026a1eb51562\",2,109529,false,\"2024-09-25 10:46:26.689\",\"2024-09-25 10:49:39.004\",NULL,NULL,false,true,\"{\\\"finalizers\\\":[\\\"otto.gptscript.ai/knowledge-file\\\"],\\\"managedFields\\\":[{\\\"apiVersion\\\":\\\"otto.gptscript.ai/v1\\\",\\\"fieldsType\\\":\\\"FieldsV1\\\",\\\"fieldsV1\\\":{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\"otto.gptscript.ai/knowledge-file\\\\\"\\\":{}}},\\\"f:spec\\\":{\\\"f:agentName\\\":{},\\\"f:fileName\\\":{}}},\\\"manager\\\":\\\"otto\\\",\\\"operation\\\":\\\"Update\\\",\\\"time\\\":\\\"2024-09-25T17:46:26Z\\\"}]}\",\"{\\\"spec\\\":{\\\"agentName\\\":\\\"a12k6wj\\\",\\\"fileName\\\":\\\"2022 Acorn Labs, LLC - Tax Return.pdf\\\"}}\",\"{\\\"fileDetails\\\":{},\\\"ingestionStatus\\\":{\\\"filepath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/9dde419b-ca64-46ad-a31d-25b4e0a1a34e/2022 Acorn Labs, LLC - Tax Return.pdf\\\",\\\"flow\\\":\\\"ingestion\\\",\\\"msg\\\":\\\"Created embedding\\\",\\\"num_documents\\\":108,\\\"phase\\\":\\\"store\\\",\\\"rootPath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/9dde419b-ca64-46ad-a31d-25b4e0a1a34e\\\",\\\"stage\\\":\\\"embedding\\\",\\\"status\\\":\\\"completed\\\",\\\"vectorstore\\\":\\\"chromem-go\\\"}}\",\"\") RETURNING `id`"

There are many threads relating to ingestion (~34) and they show the following errors:

ERROR: got (exit status 1) while running tool, OUTPUT: {"time":"2024-09-25T10:46:22.920043-07:00","level":"INFO","msg":"constraint failed: UNIQUE constraint failed: datasets.id (1555)"}

Trying to delete the agent at this time did not succeed.
Stoped and started the server .

I see the following errors show up repeatedly now:

ERRO[0042] error syncing '_t default/t1ingest-nv2sd': handler otto-controller otto.gptscript.ai/v1, Kind=Thread: [routes.go:44] run encountered an error: error, status code: 400, message: This model's maximum context length is 128000 tokens. However, your messages resulted in 128485 tokens. Please reduce the length of the messages. with error output: , requeuing
@sangee2004 sangee2004 added bug Something isn't working knowledge labels Sep 25, 2024
@thedadams
Copy link
Contributor

#66

@thedadams
Copy link
Contributor

Sorry for the confusion. The PR I linked only handles the first set of errors reported in this issue. The others are from other aspects of the ingestion process.

@sangee2004
Copy link
Author

Tested with latest build from - 4e90336505b5

When testing the same scenario mentioned in the issue , I am now seeing improvement in the same use case when testing with the fixes.
I dont see the UNIQUE constraint errors in the logs anymore.
Screenshot 2024-09-25 at 11 39 32 AM
Ingestion succeeds for some of the files. But many of them are shown in red in UI with 409 errors.

Also i am starting to see unique constraint errors in logs may be because of re-entered the agent in edit mode. Will this cause re ingestion to be triggered?

ERRO[0346] sql query error                               affected=0 caller="/Users/sangeethahariharan/go/pkg/mod/github.com/acorn-io/[email protected]/pkg/db/db.go:686" elapsed="32.209µs" error="UNIQUE constraint failed: knowledgefile.previous" sql="INSERT INTO `knowledgefile` (`kind`,`version`,`api_group`,`name`,`namespace`,`uid`,`generation`,`previous`,`create`,`created`,`updated`,`deleted`,`removed`,`garbage`,`latest`,`metadata`,`data`,`status`,`partition_id`) VALUES (\"KnowledgeFile\",\"v1\",\"otto.gptscript.ai\",\"d1933c10e5d9ca288c6c7a01f5d94b71a82f73060faca5da8231462ed624bcf8\",\"default\",\"0d181c27-0c36-428b-92c9-1135d17f776f\",2,82812,false,\"2024-09-25 10:46:23.627\",\"2024-09-25 10:48:26.954\",NULL,NULL,false,true,\"{\\\"finalizers\\\":[\\\"otto.gptscript.ai/knowledge-file\\\"],\\\"managedFields\\\":[{\\\"apiVersion\\\":\\\"otto.gptscript.ai/v1\\\",\\\"fieldsType\\\":\\\"FieldsV1\\\",\\\"fieldsV1\\\":{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\"otto.gptscript.ai/knowledge-file\\\\\"\\\":{}}},\\\"f:spec\\\":{\\\"f:agentName\\\":{},\\\"f:fileName\\\":{}}},\\\"manager\\\":\\\"otto\\\",\\\"operation\\\":\\\"Update\\\",\\\"time\\\":\\\"2024-09-25T17:46:23Z\\\"}]}\",\"{\\\"spec\\\":{\\\"agentName\\\":\\\"a12k6wj\\\",\\\"fileName\\\":\\\"AcornLabs-DO-Policy Forms_TRAV-2023.pdf\\\"}}\",\"{\\\"fileDetails\\\":{},\\\"ingestionStatus\\\":{\\\"absolute_path\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/9dde419b-ca64-46ad-a31d-25b4e0a1a34e/AcornLabs-DO-Policy Forms_TRAV-2023.pdf\\\",\\\"component\\\":\\\"index\\\",\\\"filename\\\":\\\"AcornLabs-DO-Policy Forms_TRAV-2023.pdf\\\",\\\"filepath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/9dde419b-ca64-46ad-a31d-25b4e0a1a34e/AcornLabs-DO-Policy Forms_TRAV-2023.pdf\\\",\\\"filetype\\\":\\\".pdf\\\",\\\"flow\\\":\\\"ingestion\\\",\\\"msg\\\":\\\"Creating embedding\\\",\\\"num_documents\\\":76,\\\"phase\\\":\\\"store\\\",\\\"reason\\\":\\\"duplicate\\\",\\\"rootPath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/9dde419b-ca64-46ad-a31d-25b4e0a1a34e\\\",\\\"stage\\\":\\\"embedding\\\",\\\"status\\\":\\\"starting\\\",\\\"vectorstore\\\":\\\"chromem-go\\\"}}\",\"\") RETURNING `id`"
ERRO[0346] failed to update file: Operation cannot be fulfilled on KnowledgeFile.otto.gptscript.ai "d1933c10e5d9ca288c6c7a01f5d94b71a82f73060faca5da8231462ed624bcf8": object has been modified, please apply your changes to the latest version and try again  logger=/pkg/controller/handlers/knowledge

@sangee2004
Copy link
Author

sangee2004 commented Sep 25, 2024 •
edited
Loading

The behavior mentioned in #65 (comment) is also seen when testing with lesser number of files (18 files) in my case.

I am seeing these UNIQUE constraint errors even on initial creation of agents with knowledge files.

DEBU[0658] Handling trigger [default/a1tjlx8] [otto.gptscript.ai/v1, Kind=Agent] 
DEBU[0659] Backing off [default/8c84d65a0587748d03c59ba23228a418bbe576fc41fdabb51b93d1a0f2885395] [otto.gptscript.ai/v1, Kind=KnowledgeFile] for 4.459110249s 
ERRO[0659] sql query error                               affected=0 caller="/Users/sangeethahariharan/go/pkg/mod/github.com/acorn-io/[email protected]/pkg/db/db.go:686" elapsed="34.958µs" error="UNIQUE constraint failed: knowledgefile.previous" sql="INSERT INTO `knowledgefile` (`kind`,`version`,`api_group`,`name`,`namespace`,`uid`,`generation`,`previous`,`create`,`created`,`updated`,`deleted`,`removed`,`garbage`,`latest`,`metadata`,`data`,`status`,`partition_id`) VALUES (\"KnowledgeFile\",\"v1\",\"otto.gptscript.ai\",\"7cf9ad4c212366368cc72314ffc43fb86712ba2bcc76f4535f3033e152050a5d\",\"default\",\"ed833ef9-45f0-49d8-9273-a4675a067aef\",2,89288,false,\"2024-09-25 12:03:05.918\",\"2024-09-25 12:12:06.447\",NULL,NULL,false,true,\"{\\\"finalizers\\\":[\\\"otto.gptscript.ai/knowledge-file\\\"],\\\"managedFields\\\":[{\\\"apiVersion\\\":\\\"otto.gptscript.ai/v1\\\",\\\"fieldsType\\\":\\\"FieldsV1\\\",\\\"fieldsV1\\\":{\\\"f:metadata\\\":{\\\"f:finalizers\\\":{\\\".\\\":{},\\\"v:\\\\\"otto.gptscript.ai/knowledge-file\\\\\"\\\":{}}},\\\"f:spec\\\":{\\\"f:agentName\\\":{},\\\"f:fileName\\\":{}}},\\\"manager\\\":\\\"otto\\\",\\\"operation\\\":\\\"Update\\\",\\\"time\\\":\\\"2024-09-25T19:03:05Z\\\"}]}\",\"{\\\"spec\\\":{\\\"agentName\\\":\\\"a1tjlx8\\\",\\\"fileName\\\":\\\"test1.json\\\"}}\",\"{\\\"fileDetails\\\":{},\\\"ingestionStatus\\\":{\\\"filepath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/test1.json\\\",\\\"flow\\\":\\\"ingestion\\\",\\\"msg\\\":\\\"Created embedding\\\",\\\"num_documents\\\":1242,\\\"phase\\\":\\\"store\\\",\\\"rootPath\\\":\\\"/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94\\\",\\\"stage\\\":\\\"embedding\\\",\\\"status\\\":\\\"completed\\\"}}\",\"\") RETURNING `id`"

There are some files that report status as "completed" and many others stuck in "starting" state.

{
  "items": [
    {
      "id": "80e726420082f285d3f91bfd5a7f43059a3cd6d856fe621af2444dceeff07339",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "MyFeb21.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "absolute_path": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/MyFeb21.pdf",
        "filename": "MyFeb21.pdf",
        "msg": "Ingested document",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/MyFeb21.pdf",
        "phase": "store",
        "num_documents": 3,
        "status": "completed",
        "filetype": ".pdf"
      },
      "fileDetails": {

      }
    },
    {
      "id": "5b3b847bad3012ef55e31ba45d53a40019e249e44a6d57ef10a0bfce269a7c6b",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "Holiday List_2024.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/Holiday List_2024.pdf",
        "phase": "store",
        "num_documents": 1,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "8eed35eaa539fb09743c6a3d932db1d66973dc0b77ae1ca1aecc1dbe84c625cf",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "bugs.html",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/bugs.html",
        "phase": "store",
        "num_documents": 15,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "31429e64c3231776c756da2d2e3e6ff1697a45ad27011fd9e9d8cf6ae9aee5e2",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "mystory.txt",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/mystory.txt",
        "phase": "store",
        "num_documents": 2,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "59419f9619387c908ed773118b8c27cfa51ca6041aa005693e0dbcd72400bcd0",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "Reunion-Under-The-Stars.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Created embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/Reunion-Under-The-Stars.pdf",
        "phase": "store",
        "num_documents": 3,
        "stage": "embedding",
        "status": "completed"
      },
      "fileDetails": {

      }
    },
    {
      "id": "3e6786a3254d0c1efbbde914030096e13131604674cbae4fb90e1340ece0b511",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "test.html",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "absolute_path": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/test.html",
        "filename": "test.html",
        "msg": "Ingested document",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/test.html",
        "phase": "store",
        "num_documents": 13,
        "status": "completed",
        "filetype": ".html"
      },
      "fileDetails": {

      }
    },
    {
      "id": "8c84d65a0587748d03c59ba23228a418bbe576fc41fdabb51b93d1a0f2885395",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "myjson.json",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Created embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/myjson.json",
        "phase": "store",
        "num_documents": 89,
        "stage": "embedding",
        "status": "completed"
      },
      "fileDetails": {

      }
    },
    {
      "id": "cf02b4b591af87d85da720ed452f6b86bf6a1c9256ef9cceb9b3544de5b3ac99",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "mytest.txt",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Created embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/mytest.txt",
        "phase": "store",
        "num_documents": 4,
        "stage": "embedding",
        "status": "completed"
      },
      "fileDetails": {

      }
    },
    {
      "id": "b360938f1892ed6023de1a99d18c508246ec9dca71f012e26436ad55ffed97ac",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "RTF_Sample.rtf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "absolute_path": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/RTF_Sample.rtf",
        "filename": "RTF_Sample.rtf",
        "msg": "Ingested document",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/RTF_Sample.rtf",
        "phase": "store",
        "num_documents": 4,
        "status": "completed",
        "filetype": ".rtf"
      },
      "fileDetails": {

      }
    },
    {
      "id": "36c5583ae9beab7edbfa111337e1536dcabafd29f82886b403e063630bfd2e66",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "MyFeb12.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Starting text splitter",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/MyFeb12.pdf",
        "phase": "parse",
        "num_documents": 2,
        "stage": "textsplitter",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "37b7816e1bfabe3ff024e148227f933e5729b97abdf093309e3b2336355728f5",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "transportation_data.json",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Created embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/transportation_data.json",
        "phase": "store",
        "num_documents": 43,
        "stage": "embedding",
        "status": "completed"
      },
      "fileDetails": {

      }
    },
    {
      "id": "9b95a367eb8fd8c758065260d5b7a48f421d63163bedc256d029bfcba088e1ad",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "2023q4-alphabet-earnings-release.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Created embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/2023q4-alphabet-earnings-release.pdf",
        "phase": "store",
        "num_documents": 23,
        "stage": "embedding",
        "status": "completed"
      },
      "fileDetails": {

      }
    },
    {
      "id": "0cf5eb956e651b12b7dd2a6894baa77a6ff234debab8a27747ff28dbee718d90",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "TravelPlan.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "absolute_path": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/TravelPlan.pdf",
        "filename": "TravelPlan.pdf",
        "msg": "Ingested document",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/TravelPlan.pdf",
        "phase": "store",
        "num_documents": 27,
        "status": "completed",
        "filetype": ".pdf"
      },
      "fileDetails": {

      }
    },
    {
      "id": "9dd012ac5315f95894eee591dfdeccab309218bfb8496e5899e75638552f50ee",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "Insurance_Handbook_20103.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/Insurance_Handbook_20103.pdf",
        "phase": "store",
        "num_documents": 391,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "8870d9d71f1615d4be8ceb0f8bfc9191167902eda15c5de13a51318ebb202a92",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "1000-Ways-to-Make-1000-Dollars.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/1000-Ways-to-Make-1000-Dollars.pdf",
        "phase": "store",
        "num_documents": 834,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "4a605295812826548fe71c81206c07f2717ee94fb2d3999c577d882667c24b66",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "dictionary.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/dictionary.pdf",
        "phase": "store",
        "num_documents": 2568,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "e7cca91b139f304215023d1e0b2177409e7c2b3f790c67c1897de0da019d24dd",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "essay.pdf",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/essay.pdf",
        "phase": "store",
        "num_documents": 408,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    },
    {
      "id": "7cf9ad4c212366368cc72314ffc43fb86712ba2bcc76f4535f3033e152050a5d",
      "created": "2024-09-25T12:03:05-07:00",
      "fileName": "test1.json",
      "agentID": "a1tjlx8",
      "ingestionStatus": {
        "msg": "Creating embedding",
        "flow": "ingestion",
        "rootPath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94",
        "filepath": "/Users/sangeethahariharan/Library/Application Support/otto/workspaces/5d20fb90-41d8-4de5-b912-650df63daa94/test1.json",
        "phase": "store",
        "num_documents": 1242,
        "stage": "embedding",
        "status": "starting"
      },
      "fileDetails": {

      }
    }
  ]
}

@sangee2004
Copy link
Author

https://github.com/gptscript-ai/otto-ui/pull/40

@thedadams
Copy link
Contributor

#67

@thedadams
Copy link
Contributor

The backend change for this has been merged.

@StrongMonkey StrongMonkey mentioned this issue Sep 26, 2024
Merged
@cjellick
Copy link
Contributor

cjellick commented Nov 4, 2024

please retest

@sangee2004
Copy link
Author

This issue is not seen any more when testing with 500+ knowledge files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working knowledge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cjellick @thedadams @StrongMonkey @sangee2004