[issue in need of revisiting] Retrieval Augmentation / Memory

[Pwuts]

🚧 Actual state of work in progress

In order to effectively utilize RAG for long-term memory in AutoGPT, we need to find or create data points during the think-execute cycle which we can use to query a vector DB and enrich the context. Our current vision is that the best way to realize these data points is to implement structured planning/task management. This is tracked in #4107.

Problem

Vector memory isn't used effectively.

Related proposals

• Use chatgpt-retrieval-plugin's datastore for memory + permanent memory #2058

Related issues (need deduplication)

• Memory Feature seems to not work on gpt3only mode #623
• Memory seeding does not work with local memory caching #2072
• Failing to use pre-seeded data and/or chunk a large JSON file #2076
• Relevant memory is essentially a duplicate of the conversation #2232
• Add Recency and Importance for Memory Retrieval #2893
• Theres no meta data in the memories #3451

🔭 Primary objectives

• Robust and reliable memorization routines for all relevant types of content 🏗️ 1+3
  This covers all processes, functions and pipelines involved in creating memories. We need the content of the memory to be of high quality to allow effective memory querying and to maximize the added value of having these memories in the first place.
  TL;DR: garbage in, garbage out -> what goes in must be focused towards relevance and subsequent use.

  • Webpages 🏗️ 1
  • Text files 🏗️ 1
  • Documents 🏗️ 3
    • Text file loaders #3031
    • LangChain has various document loaders: see docs
  • Code files 🏗️ 3
    • OpenAI cookbook > code search
    • https://github.com/BillSchumacher/SumPAI
    • https://pypi.org/project/hfcca/ -- https://github.com/terryyin/lizard
  • Lists
  • Tables

• Good memory search/retrieval based on relevance 🏗️ 1
  For a given query (e.g. a prompt or question), we need to be able to find the most relevant memories.

  Must be implemented separately for each memory backend provider:

  • JSON
  • Redis (blocked by Add support for KNN vector similarity search redis/redis-om-python#513)
  • Milvus

  (The other currently implemented providers are not in this list because they may be moved to plugins.)

• Effective LLM context provisioning from memory 🏗️ 2
  Once we have an effective system to store and retrieve memories, we can hook this into the agent loop. The goal is to provide the LLM with focused, useful information when it is needed or useful for the next step of a given task.

  • Determine what context data we can best use to find relevant memories. This is non-trivial because we are looking for information that is relevant to the next step before that step has been specified.

🏗️ Pull requests

1. Vector memory revamp (part 1: refactoring) #4208
   Applies refactoring and restructuring to make way for bigger improvements to the memory system
2. Agent loop v2: Planning & Task Management (part 1: refactoring) #4799
   Restructures the Agent to make extension and modification easier
3. [to be created], implements effective use of available memory in the agent loop
4. [to be created], adds memorization routines for more content types
5. [to be created], adds visualizations for functionality involving embeddings

🛠️ Secondary todo's

• Refactoring 🏗️ 1
• Improving text summarization 🏗️ 1
• Implementing code summarization/description/search 🏗️ 3
  (see also Code files under 🔭 Primary objectives)

✨ Tertiary todo's

• Visualizations 🏗️ 4
  • Embeddings
  • Similarities
  • Memory state (graph?)

📝 Drawing boards

These boards contain drafts, concepts and considerations that form the basis of this subproject. Feel free to comment on them if you have ideas/proposals to improve the workflows or schematics.
If you are curious and have questions, please ask those on Discord.

• Agent workflow v2 - redesign 🚧
• Memory system subroutines
• Agent workflow (current/old)

Related

• LMQL could be a significant improvement (tracked in issue LMQL #3460, cc @AbTrax)

🚨 Please no discussion in this thread 🚨

This is a collection issue, intended to collect insights, resources and record progress. If you want to join the discussion on this topic, please do so in the Discord.

[Pwuts]

Related PRs:

• Fix the maximum context length issue by chunking #3222

[Wladastic]

I will tinker with this shortly:
#4467

[W3Warp]

Sorry I am new to this, do I post this here or somewhere else?
W3Warp#4 (comment)

[Pwuts]

@W3Warp depends what you want to achieve

[W3Warp]

@W3Warp depends what you want to achieve

Help with the issue.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.