[Meta] Improving queries over archival memory

[sarahwooders]

Currently, archival memory works by loading in any attached data sources and saves memories from the agent into a single table (representing the agent's archival memory), which has a "text" and "embedding" columns.

Customizable queries

• The agent should be able to combine embedding search with:
  • Text/date range based search Add hybrid/text search for archival memory #362
  • Retrieving entire documents
  • Filtering by specific data sources Not possible to prompt memgpt to search a specific archival store when multiple have been attached #395
• This will be abled in part by customizable presets to define custom queries Configurable presets #339

Better performance

• The retriever could access data without copying it into the agent's archival storage table by searching across multiple tables

Other issues

• Archival search only looks for top-5 results and often does not attempt to search further
  • Perhaps agent is never updating the offset?
• The chunk size is very small (300) and may need to be tuned
  • We may want to include an option of embedding entire documents

Related Issues

• Configurable presets #339
• Not possible to prompt memgpt to search a specific archival store when multiple have been attached #395
• Add hybrid/text search for archival memory #362
• Running /attach is very slow for large datasets #371

[github-actions]

This issue has been automatically closed due to 60 days of inactivity.