MemGPT + AutoGen

[cpacker]

Integrate MemGPT into AutoGen!

Instructions on how to connect MemGPT to AutoGen can be found on our docs page.

🙋 Need help with local MemGPT + AutoGen? Check Discord!

If you need help visit our Discord server and post in the #support channel.

You can also use this GitHub discussions page, but the Discord server is the official support channel and is monitored more actively.

[tomsib2001]

Note that the folks at AutoGen are also referencing this project: microsoft/autogen#156 (comment)

[mberman84]

This would be incredible!

[scottiesan]

I love this work!!

[console-1]

This would truly be a game changer and in turn possibly creating actual super agents. Any idea how far along the roadmap this might be @cpacker ? Not rushing at all, just excited to see how you and the squad go about it. I have been trawling the web high and low to find a solution to integrate MemGPT into AutoGen but have come up emptier than my wallet around the 5th of each month. Regardless of when I still wanted to thank you and the contributors for creating an amazing project. Not all heroes wear capes.

[QZGao]

I'm trying on my own to restructure the AgentAsync following the AutoGen's approach on multi-agents, but there is a place where I couldn't figure out why it is written that way:

Line 78 of main.py

Isn't user_input = None? Why does it become a file name? Is it safe to remove user_input in main() (if I somehow make user_input into a class field of UserAgent and replace the while loop with something else)?

[cpacker]

@QZGao 🐞 caught thanks for that

[briancunning]

This would be amazing. It was the first thing I thought of when I saw this repo.

[cpacker]

We just added some basic AutoGen integration! #64

We're still working on cleaning up the workflow (adding better MemGPT-AutoGen constructors that better match the config file style of AutoGen), but since it seems like a lot of people want AutoGen support we thought it's best to release what we have for now and add to it as we go.

Current caveats

• You have to specify OPENAI_API_KEY still (with set or export), you can't reply on config_list to set it for you
• You have to manually specify the persona+user, persona should come from the AutoGen system prompts
• Probably many more, so let us know what we need to add next

Example using MemGPT agent inside AutoGen

Updated 10/26: see the example on the main post: #65 (comment)

[cpacker]

With the current integration code, you'd do something like this:

memgpt_agent = presets.use_preset(
  presets.DEFAULT,
  'gpt-4',
  personas.get_persona_text(personas.DEFAULT),  # change this line!
  humans.get_human_text(humans.DEFAULT),
  interface,
  persistence_manager
)

memgpt_agent = presets.use_preset(
  presets.DEFAULT,
  'gpt-4',
  'Creative in software product ideas.',  # change this line!
  humans.get_human_text(humans.DEFAULT),
  interface,
  persistence_manager
)

But looking to improve the integration to make this smoother in the future!

[mlops-v2-gs]

Thanks for this amazing library. I've tried to use the suggested code and getting an error.
ImportError: cannot import name 'printd' from partially initialized module 'memgpt.utils' (most likely due to a circular import) (/workspaces/Autogen/memgpt/utils.py)
Any suggestions on how to fix that? I've tried this on exact same code above as well as one my own agent and ended getting the same.

[whithr]

@mlops-v2-gs there's a circular dependency, to just fix for now you can remove the "from memgpt.utils import printd" in the memgpt/interface.py file and copy paste the imported printd function into that interface.py file.

def printd(*args, **kwargs): if DEBUG: print(*args, **kwargs)

[cpacker]

this should be fixed in the latest commit

[mlops-v2-gs]

Thanks a lot. Worked like a charm.

[console-1]

This is truly amazing @cpacker !!

[Peter2023AI]

I don’t have the expertise to directly assist in this programming endeavor, but I wanted to know if there was a way I could contribute to this by using ChatGPT4. The following is my prompt to ChatGPT4 and it’s response in regards to combining MemGPT+AutoGen+RWKV (I only chose RWKV because that LLM has a one click install and includes a API). It would be great to have MemGPT+AutoGen to be able to run locally. I’d pay a monthly subscription fee for code updates to something like this, if it was making decent headway with improvements each month. I’m not sure if any of this feedback of information is useful or not, but wanted to toss this info out there.

My Prompt to ChatGPT4 + Web Browsing:
Please read the content on https://github.com/cpacker/MemGPT and #65 and on https://github.com/microsoft/autogen and create a workflow to successfully string together code snippets to combine these together, and run the resultant AI from a Windows desktop computer running a local copy of a large language model called RWKV which has an API https://github.com/josStorer/RWKV-Runner? Please also provide the ChatGPT4 prompts necessary to accomplish this goal if possible, and if you hit the character response limit to reply to me, please let me know, and I will prompt you to continue.

ChatGPT4 Response:
Creating a unified workflow by stringing together code snippets from MemGPT, AutoGen, and a Large Language Model (LLM) like RWKV involves several steps. The workflow may look like this:
1. Setup Environment:
o Ensure your Windows desktop computer has Python installed.
o Clone the necessary repositories: MemGPT, AutoGen, and RWKV-Runner.
o Install all the dependencies required by MemGPT, AutoGen, and RWKV-Runner as per their respective documentation.
2. Understand Individual Components:
o Go through the codebase and documentation of MemGPT, AutoGen, and RWKV-Runner to understand their functionalities and architectures.
o Understand how MemGPT manages memory for LLMs, how AutoGen facilitates large language model applications, and how RWKV-Runner interacts with the RWKV LLM.
3. Code Integration Planning:
o Identify the key components or code snippets in MemGPT that are crucial for memory management.
o Identify the parts of AutoGen where these memory management snippets could be integrated.
o Understand how RWKV-Runner API can be called for interacting with the RWKV LLM.
o Plan how these components will interact with each other.
4. Code Integration:
o Create a new project directory for the integrated application.
o Copy the necessary code snippets from MemGPT and integrate them into the relevant parts of AutoGen.
o Modify the AutoGen code to use the RWKV-Runner API for interacting with the RWKV LLM.
o Ensure that the memory management features from MemGPT are correctly utilized by AutoGen when interacting with the RWKV LLM.
5. Testing and Debugging:
o Thoroughly test the integrated application to ensure that memory management is working as expected, and the RWKV LLM is correctly utilized.
o Debug and fix any issues that arise during testing.
6. Running the Application:
o Once the integrated application is tested and debugged, run it on your Windows desktop computer.
o Monitor the application for any errors or issues and resolve them as needed.
7. ChatGPT-4 Prompts:
o There isn’t a direct way to create ChatGPT-4 prompts for this workflow as it largely involves coding and integration tasks. However, you might create custom scripts or applications that utilize ChatGPT-4 for specific tasks within this workflow, like generating code snippets or providing explanations for specific APIs or code logic.
8. Documentation:
o Document the integration process, any challenges encountered, and how they were resolved.
o Create a README file or documentation explaining how to run the integrated application, what it does, and any other necessary information.

This workflow provides a structured approach to integrating MemGPT, AutoGen, and the RWKV LLM to create a unified AI application. The actual process may involve additional steps or modifications based on the specifics of the codebases and the RWKV LLM.
--- End of ChatGPT4's Response ---

[PotatoKilla]

i know this is something im probably doing wrong but im getting import memgpt.autogen.memgpt_agent as memgpt_autogen ModuleNotFoundError: No module named 'memgpt.autogen'; 'memgpt' is not a package

[QZGao]

Try install it from PyPI, or clone from GitHub (and put it under the same directory) first.

[Sounart]

I keep getting the loop like the example above where the bots just say “heres that request again” a bunch of times until they hit the limit. Anyone know why this is happening?

[cpacker]

This should be fixed in a patch that went live yesterday!

If you did a local install, just git pull then pip install -e .. If you did a pip install, just do pip install pymemgpt -U.

[xoich]

Hi, in the Readme you say you are willing to support other server type besides webui: Supporting the llama.cpp server would be nice. thank you for your great work!

[cpacker]

Did you mean to post this in #67 ?

[cpacker]

An AutoGen update: we just updated the AutoGen MemGPT agent to:

• Fix a few bugs (MemGPT AutoGen agents should now behave better, before they might say things like "the user is repeating themselves")
• Make it a lot easier/cleaner to create MemGPT AutoGen agents

The recommended way to create a MemGPT AutoGen agent now is to use the create_autogen_memgpt_agent function:

import os
import autogen
from memgpt.autogen.memgpt_agent import create_autogen_memgpt_agent

def your_autogen_script():
    ...

    coder = create_autogen_memgpt_agent(
        "MemGPT_coder",
        persona_description="I am a 10x engineer, trained in Python. I was the first engineer at Uber (which I make sure to tell everyone I work with).",
        user_description=f"You are participating in a group chat with a user ({user_proxy.name}) and a product manager ({pm.name}).",
        # extra options
        # interface_kwargs={"debug": True},
    )

    groupchat = autogen.GroupChat(agents=[user_proxy, pm, coder], messages=[], max_round=12)

We've updated the example (that was previously in a discussion comment, now in the repo code) to use this new style of constructor, you can see the example here.

[gpilleux]

Hello @cpacker awesome integration with Autogen!
Been reading the code and can't figure out how to use the archival_storage_files from MemGPT with the create_autogen_memgpt_agent function.
It seems that your main.py handles many cases regarding preloading a knowledge base.
I have also seen that the create_autogen_memgpt_agent handles the persistence_manager argument.
How would the code look if I wanted the coder agent in your example to have a preloaded knowledge base, for example a git repository or a local directory with many files in it?

[cpacker]

This is a great point and something that's currently lacking in the AutoGen implementation.

We will add this ASAP.

[Peter2023AI]

It’s my understanding that currently all of the models (that are specified) currently load at start up, which limits how many specialized models can be used because of memory limitations. What if an option were added to the code location where each model is defined that would specify one of two options.

Option 1: The model is loaded at startup.
Option 2: The model is only loaded when called upon and then unloaded when its task is completed.

This way someone could mix and match which models load at startup and stay loaded in memory, while other models are loaded and unloaded on demand based on a specific task.

If the above is implemented, would it also then make sense have separate task specific “archival_storage_files” repositories, besides a global “archival_storage_files” location. If each model is task specific, and if each model had it’s own task specific “archival_storage_files” repository, then when a model is loaded or unloaded from memory its corresponding knowledge base of associated data (“git repository or a local directory with many files in it”) could also get loaded or unloaded from memory.

If both of the above were implemented, the user community could perhaps start contributing to the task specific repositories with the contents of the repositories remaining transparent to end users. We could even have validated and non-validated repositories.

Fantastic job by both the MemGPT and AutoGen teams!!!

[jms90h5]

This particular capability is critically important for one of my "ask the docs" use cases". It seems that all the archival storage interaction is now only in the "legacy" interface. I realize you have a ton of things on your plate but would it help to open an issue for this capability so it can at least be tracked in your long priority list?

[khawajaJunaid]

Hi @cpacker , I wanted to confirm if we still dont have support for memgpt knowledge base like files directory or has it been handled. I could not find much documentation on this part, would appreciate the help.

[NZ369]

Does anyone else keep getting this error?
return asyncio.run(
File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/ana/.local/lib/python3.10/site-packages/memgpt/autogen/memgpt_agent.py", line 148, in _a_generate_reply_for_user_message
user_message = new_messages[0]
IndexError: list index out of range

[cpacker]

What's the context of this error? As in what kind of environment was the MemGPT AutoGen agent in when this error got triggered.

This is a bug we're going to patch in #143, but this bug should only happen when the MemGPT agent gets run on a message history (from the AutoGen conversation manager), but all the messages in the message history are old (there are no new messages).

[wdressler]

Might be related to NZ369 issues, but running the script, I get this error: RuntimeError: asyncio.run() cannot be called from a running event loop.

After researching, I found that running it in Colab may be the issue. There is an alt library called 'nest-asyncio' that could resolve it. But I'm not sure how to test that. (Still learning Python)

[cpacker]

Can you try updating the install? pip install pymemgpt -U, or git pull and pip install -e . if you're installing from source.

This might be an old bug that's been patched.

[wdressler]

I tried that and even tried !pip install pyautogen --no-cache --upgrade

But I got the same error.

[wdressler]

Colab AI explained the error this way:

The code is failing because the run_chat() method in the GroupChat class is calling the generate_reply() method on the MemGPT_coder agent, which is an asynchronous function. However, the run_chat() method is not running inside an asynchronous context, so the generate_reply() method cannot be called.

To fix this, you can either make the run_chat() method asynchronous, or you can call the generate_reply() method inside an asynchronous context. For example, you could use the asyncio.run() function to run the generate_reply() method asynchronously.

I just don't know how to implement what it is saying.

[wdressler]

Do you want all of the frames? It is able to run and generate the first response, but then crashed right after the response.

Error:

User_proxy (to chat_manager):

I want to design an app to make me one million dollars in one month. Yes, you heard that right.

Product_manager (to chat_manager):

Ideation 1: The "Luxury Experience" App
Your app could be aimed at high-end clientele, focusing on unique, opulent experiences like personal jet rentals, mega yacht bookings, luxury car rentals, private island rentals, and rare gourmet experiences. With a high commission rate and a niche but willing client base, you could reach a million dollars in bookings (and thus income for you) quickly.

Ideation 2: Fitness App with Celebrity Trainers
With the current obsession for health and wellness, you could create a unique fitness app featuring well-known, celebrity trainers. Offer expensive subcription plans and personalized training courses for affluent clients.

Ideation 3: High-End Art Auctioning App
An app that provides a platform to buy and sell high-end art pieces could be a huge moneymaker. This app could connect buyers and sellers, curate collections and auctions, and even offer investment advice in the art world.

Ideation 4: Business Networking App
An app that brings together entrepreneurs, investors, venture capitalists and industry experts could be highly valuable and lucrative. This high-end networking and knowledge-sharing platform could be monetized through membership fees and premium services.

However, it's important to remember that even though your product can theoretically generate a million dollars, its successful implementation depends on many factors like market value, competition, marketing strategy and so on. Additionally, even successful apps typically need at least a few months to gain the user base required to earn significant income.

RuntimeError Traceback (most recent call last)
in <cell line: 64>()
62
63 # Begin the group chat with a message from the user
---> 64 user_proxy.initiate_chat(
65 manager,
66 message="I want to design an app to make me one million dollars in one month. Yes, your heard that right.",

7 frames
[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/conversable_agent.py](https://localhost:8080/#) in initiate_chat(self, recipient, clear_history, silent, **context)
529 """
530 self._prepare_chat(recipient, clear_history)
--> 531 self.send(self.generate_init_message(**context), recipient, silent=silent)
532
533 async def a_initiate_chat(

[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/conversable_agent.py](https://localhost:8080/#) in send(self, message, recipient, request_reply, silent)
332 valid = self._append_oai_message(message, "assistant", recipient)
333 if valid:
--> 334 recipient.receive(message, self, request_reply, silent)
335 else:
336 raise ValueError(

[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/conversable_agent.py](https://localhost:8080/#) in receive(self, message, sender, request_reply, silent)
460 if request_reply is False or request_reply is None and self.reply_at_receive[sender] is False:
461 return
--> 462 reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
463 if reply is not None:
464 self.send(reply, sender, silent=silent)

[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/conversable_agent.py](https://localhost:8080/#) in generate_reply(self, messages, sender, exclude)
779 continue
780 if self._match_trigger(reply_func_tuple["trigger"], sender):
--> 781 final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
782 if final:
783 return reply

[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/groupchat.py](https://localhost:8080/#) in run_chat(self, messages, sender, config)
162 speaker = groupchat.select_speaker(speaker, self)
163 # let the speaker speak
--> 164 reply = speaker.generate_reply(sender=self)
165 except KeyboardInterrupt:
166 # let the admin agent speak if interrupted

[/usr/local/lib/python3.10/dist-packages/autogen/agentchat/conversable_agent.py](https://localhost:8080/#) in generate_reply(self, messages, sender, exclude)
779 continue
780 if self._match_trigger(reply_func_tuple["trigger"], sender):
--> 781 final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
782 if final:
783 return reply

[/usr/local/lib/python3.10/dist-packages/memgpt/autogen/memgpt_agent.py](https://localhost:8080/#) in _generate_reply_for_user_message(self, messages, sender, config)
120 config: Optional[Any] = None,
121 ) -> Tuple[bool, Union[str, Dict, None]]:
--> 122 return asyncio.run(
123 self._a_generate_reply_for_user_message(
124 messages=messages, sender=sender, config=config

[/usr/lib/python3.10/asyncio/runners.py](https://localhost:8080/#) in run(main, debug)
31 """
32 if events._get_running_loop() is not None:
---> 33 raise RuntimeError(
34 "asyncio.run() cannot be called from a running event loop")
35

RuntimeError: asyncio.run() cannot be called from a running event loop

[vivi]

Try (before the memgpt autogen example code):

%pip install nest_asyncio

import nest_asyncio
nest_asyncio.apply()

tl;dr colab is running an outside async loop, and we use asyncio.run() to make a synchronous version of generate_reply.

TODO: look into a more elegant way to fix this. --> should be fixed by #157

[cpacker]

For reference, this is what the example looks like if you turn auto-reply for the user off and prod it to actually start coding:

User_proxy (to chat_manager):

I want to design an app to make me one million dollars in one month. Yes, your heard that right.

--------------------------------------------------------------------------------
Product_manager (to chat_manager):

App Idea: "Luxe Auction - The Luxury Auctioning App"

This app is targeted at high-net-worth individuals across the globe. It hosts auctions of high-end, rare, antique, and luxury items. These include anything from rare art pieces, first edition books, ancient artifacts, rare wines, luxury cars, real estate, and more. The app will also offer a secure, digital platform on which these transactions could safely take place.

How it makes money:

1. Listing fees: Similar to eBay, every listing on the site could come with a fee that would depend on the price of the good or service being sold.

2. Selling fees: Aside from the listing fees, the app could take a portion of the selling price as an additional fee.

3. Premium Subscription: Users who subscribe to the premium version of the app get perks such as early access to new listings, premium customer service, and more.

4. In-app purchases: The app could feature an online currency used to make bidding faster and more efficient.

5. Sponsorship and Partnerships: Luxury brands can sponsor the app or certain auctions, and in turn, the app can promote these brands.

Remember, patience is key in making an app successful. While the app might not make you one million dollars in the first month, with a solid marketing strategy and time, you will be able to reach your goal.

--------------------------------------------------------------------------------
MemGPT_coder (to chat_manager):

That's a fascinating concept, Product_manager. Given my experience in engineering Uber's initial stages, I can't stress enough the importance of solid backend infrastructure and ironclad security. Our target clientele and the nature of the transactions demand the best we can offer in both aspects.

--------------------------------------------------------------------------------
Provide feedback to chat_manager. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: OK - let's get to work on the actual idea. Start now.
User_proxy (to chat_manager):

OK - let's get to work on the actual idea. Start now.

--------------------------------------------------------------------------------
MemGPT_coder (to chat_manager):

Alright, let's jump right in. We need to outline the core functionalities: a reliable bidding system, a secure payment process, and an item verification system. Lastly, the user experience should be unparalleled. As the main engineer, I will start by creating a scalable and efficient system architecture.

--------------------------------------------------------------------------------
Product_manager (to chat_manager):

Great, Engineer. As the product manager, my responsibility is to establish the product roadmap and ensure it aligns with our business strategy. We aim to create value for our customers, especially the high-class society. I will focus on:
 
1. Market analysis: Understanding the needs of our target customers and shaping our app to meet those requirements.
2. Pricing Strategy: Defining our app fees, subscription plans, partnership rates, etc.
3. Collaboration: Working with UX/UI designers to enhance customer experience and journey through our app.
4. Launch Strategy: Planning the right approach to introduce our app to the market.
5. Partnerships & Sponsors: Orchestrating alliances with luxury brands and auction houses to offer exclusive items.

Let's work together in sync to make this a success.

--------------------------------------------------------------------------------
Provide feedback to chat_manager. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: Please start doing your 10x coding. No wasting any more time.
User_proxy (to chat_manager):

Please start doing your 10x coding. No wasting any more time.

--------------------------------------------------------------------------------
MemGPT_coder (to chat_manager):

Understood, time to kick off the 10x coding. I'll get started with the system architecture. Remember, even in fast-paced environments, planning is key to prevent technical debt. I'll focus on creating an efficient system flow first.

--------------------------------------------------------------------------------
Product_manager (to chat_manager):

Absolutely, Engineer. As the Product Manager, I'll work in parallel to create a comprehensive market analysis report and define the UX/UI specifics for the application. We'll stay in sync to ensure all our efforts are aligned. All hands on deck!

--------------------------------------------------------------------------------
Provide feedback to chat_manager. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: Your next message should be code. Start now.
User_proxy (to chat_manager):

Your next message should be code. Start now.

--------------------------------------------------------------------------------
MemGPT_coder (to chat_manager):

Sure thing. Here's a high-level starting point, using a Python Flask server as an example:

```python
def initialize_app():
    app = Flask(__name__)

@app.route("/item/<item_id>")
def get_item(item_id):
    item = Database.get_item(item_id)  # pseudo-code, actual implementation might differ
    if item is None:
        abort(404)
    return item

@app.route("/place_bid", methods=["Post"])
def place_bid():
    # Data validation and error handling to be implemented.
    item_id = request.form['item_id']
    user_id = request.form['user_id']
    bid = request.form['bid']
    # Method to update bid to be implemented.
    item = Database.update_bid(item_id, user_id, bid)  # pseudo-code, actual implementation might differ
    return item

if __name__ == "__main__":
    initialize_app()
    app.run()

Remember, this is primarily high-level; we'd need to integrate database management, error handling, and user verification, among other concerns.

[yiranwu0]

What's the difference of using MemGPT_coder instead of a vanilla agent coder? Is there an example to show what is going on inside memGPT (like summarize, retrieve data) and why it makes it better?

[cpacker]

Hey @kevin666aa - if you run the MemGPT examples in our README you'll see the full message traces that MemGPT generates. You can also check out the MemGPT paper or one of the MemGPT YouTube videos to understand what MemGPT is doing under the hood to extend context length: https://www.youtube.com/watch?v=QQ2QOPWZKVc

[cpacker]

(In these examples, the inner monologue and function calls are hidden to simplify what the outputs look like to the other AutoGPT agents)

[yiranwu0]

Is there an option to turn on the logging so that we know what is going on when running autogen groupchat?

[vivi]

You can set interface_kwargs and pass it in like so here.

[nuaimat]

@cpacker any way of passing something like
model = "dolphin-2.1-mistral-7b"
to create_autogen_memgpt_agent

this might be used to run MemGPT and AutoGen on a local LLM.
today it gives:
openai.error.InvalidRequestError: functions is not supported.
i can provide full error log if needed.

[nmx0]

I got it to work:

config_list_memgpt = [
    {
        "model": "dolphin-2.1-mistral-7b",
    },
]

llm_config_memgpt = {"config_list": config_list_memgpt, "seed": 42}

coder = create_memgpt_autogen_agent_from_config(
    "MemGPT_coder",
    llm_config=llm_config_memgpt,
    system_message=f"I am a 10x engineer, trained in Python. I respond with great code "
                    f"My responses are mostly code. "
                    f"You are participating in a group chat with a user ({user_proxy.name}) "
                    f"and a product manager ({pm.name}). You should provide code",
)

for me it loads proper wrapper

[Jirito0]

@nmx0 I attempted implementing these changes, exactly as you show here, but was unsuccessful.

Were there any other additional changes that you made, or would you care to share your full code?

[vivi]

@jtuck41 @nmx0 Integrated what @nmx0 was doing and made it work with the most recent memgpt build here: https://github.com/cpacker/MemGPT/tree/main/memgpt/autogen

[anir-kent]

Hi @cpacker,
I am thinking to use this for generating the explanation for large code files (> context window) or the complete app code (multiple code files). Checked the paper but appears that the current compression technique you are using may not be effective for Code. Could you please share your thoughts on this?

[anjaleeps]

@cpacker is function calling currently not supported with MemGPTAgents?

[loginforai]

I've tried to work this out, but sadly when I have given the agent the ability to call functions, it lost its whole MemGPT-powers. I'm hoping someone else who is more skilled can figure this out.

[anjaleeps]

I get an error if the first message from the agent becomes a function call. Did you encounter something like that too?

[khawajaJunaid]

@cpacker i connected a postgres database to the autogen memgpt agent but the issue is it only loads the embeddings and I cannot find where the chat memory is stored which becomes crucial for context retrieval. Does the integration allow us to store our conversations in the connected database or is it even using the context

[khawajaJunaid]

@cpacker @vivi Can anyone tell me what the code would look like if we are running two Autogen_memgpt agents, I am following the one in the example for the group chat and if i add another

 coder = create_memgpt_autogen_agent_from_config(
        "MemGPT_coder",
        llm_config=llm_config_memgpt,
        system_message=f"I am a 10x engineer, trained in Python for writingscripts. I was the first engineer at Uber "
        f"(which I make sure to tell everyone I work with).\n"
        f"You are participating in a group chat with a user ({user_proxy.name}).",
        interface_kwargs=interface_kwargs,
    )
    coder.attach("embedded")  # See https://memgpt.readthedocs.io/en/latest/autogen/#loading-documents
    
    artist = create_memgpt_autogen_agent_from_config(
        "MemGPT_artist",
        llm_config=llm_config_memgpt,
        system_message=f"I am a pro artist "
        f"You are participating in a group chat with a user ({user_proxy.name} and {coder.name}) and contribute to the chat.",
        interface_kwargs=interface_kwargs,
    )

but the latter agent does not take part in the conversation, am I missing something

[vrda23]

Is there a new memgpt update to make it compatible with the latest autogen version (0.2.0b5)?

[ocarlisle]

Does anyone have a working example of how to use this for document QA with Autogen? I've been looking for hours and I've be unable to find a single example of this, which is strange because it seems like it would be the most common application of this library. Im trying to use MemGPT inside of my AutoGen script for enhanced context for QA. I've got MemGPT up and running fine and I have Autogpt working fine with RAG, but I want to use MMR via MemGPT. I have yet to see anyone post a working example of this. Does anyone have one I can look at to see how you managed it?

[Peter2023AI]

Method towards AGI with MemGPT and AutoGen

Note: The following is just some quick examples of the reletively rapid improvments that could be made to the MemGPT/AutoGen hybrid software solution that would enable dramatic improvements to requests to complete goals, tasks, or problems. Function calls would need to be aligned to “ChatGPT Response 2”, which could be streamlined and expanded upon.

Question 1:

If an LLM is instructed to read, write, understand, and update a text document every time the LLM interacts with a user, what English text could be stored within the text document that would provide logic and reasoning skills to the LLM that would also allow it to expand upon that logic and reasoning skills?

ChatGPT Response 1:

The concept of having a large language model (LLM) like GPT-4 read, write, understand, and update a text document in each interaction to improve its logic and reasoning skills is an interesting one. While LLMs don't "learn" in the traditional sense from each interaction (as their learning happens during the training phase on a fixed dataset), there are certain types of texts that can be useful in this scenario. Here are some suggestions:

1. Philosophical Texts: Works that cover logic, ethics, epistemology, and other branches of philosophy. These texts often involve complex reasoning and argumentation, which can serve as good examples for developing reasoning skills.
2. Scientific Literature: Papers and articles from fields like physics, biology, and computer science. These documents often include logical structures, hypothesis testing, and evidence-based reasoning.
3. Mathematical and Logical Puzzles: Problems and puzzles that require logical reasoning and problem-solving skills. These can include classic logic puzzles, mathematical problems, and other brainteasers.
4. Critical Thinking and Argumentation Texts: Books or articles that teach critical thinking skills, argumentation techniques, and logical fallacies. These texts can help the model understand how to construct and deconstruct arguments effectively.
5. Historical Texts and Case Studies: Detailed analyses of historical events or case studies from various fields. These texts often involve understanding complex situations, drawing inferences from limited information, and making connections between different events.
6. Literary Analysis: Works of literary criticism and analysis, which require understanding themes, motives, and symbolic representations in various forms of literature.
7. Legal Documents and Case Law: Legal texts, including laws, legal cases, and judgments, which involve complex reasoning and interpretation of legal language and principles.
8. Interactive and Recursive Texts: Texts designed specifically for the purpose of teaching reasoning, such as interactive fiction or texts that encourage the reader to make predictions and then provide feedback.
9. AI and Machine Learning Research Papers: To understand the limits and capabilities of AI, including discussions on AI ethics, safety, and potential.
10. Cross-Disciplinary Research: Texts that integrate knowledge from multiple fields, encouraging holistic thinking and the ability to connect disparate concepts.
    However, it's important to note that while exposing an LLM to such texts might provide a rich context for its responses, the LLM doesn't inherently "expand" its logic and reasoning skills like a human learner would. Its abilities are constrained by its training and the algorithms that govern its function. The LLM can simulate understanding and reasoning based on patterns it has learned, but it doesn't develop these skills autonomously.

Question 2:

Please thoroughly understand the following and work through your answer step by step. A human student can read and follow instructions or processes previously written by a human teacher, which enable the human student to solve previously unseen problems by the human student. These instructions are reusable in different situations and can be combined with additional instructions or processes from the human teacher, enabling the human student to continue to develop unique processes to solve new goals, problems, or tasks once the goal or problem is understood. This previous situation should be able to be replicated for artificial intelligence. If a large language model (LLM) is combined with MemGPT (enabling a long-term memory database and ability to do function calls to that memory) in combination with AutoGen (enabling multiple agents to work together which replicates a human having internal dialog and thought processes) then if the long-term memory database contains a set of general methods for hypothesis testing, evidence-based reasoning, logical reasoning, problem-solving skills, drawing inferences from limited information, making connections between different processes, etc., then each time a user provides a new goal, problem, or task the LLM can search through the long-term memory database for the best matching combination of hypothesis testing, evidence-based reasoning, logical reasoning, problem-solving skills, etc. processes in order to provide the user with the best solution to the goal, problem, or task. What specific text based English instructions can be written to store within the LLM’s long-term memory database that the LLM can follow to provide a framework for hypothesis testing, evidence-based reasoning, logical reasoning, problem-solving skills, drawing inferences from limited information, making connections between different processes, etc.? These textual instructions/processes should be written as short and concise instructions/processes with accurate label identifiers, which will enable the LLM to quickly use function calls to locate the correct combination of instructions/processes needed to best complete the requested goal, problem, or task. Keep in mind that these instructions/processes must be able to be applied to a broad range of situations.

ChatGPT Response 2:

The idea of replicating human-like learning and problem-solving in a large language model (LLM) involves creating a framework where the model can access and apply a set of general methods for reasoning and problem-solving. The integration of technologies like MemGPT and AutoGen can potentially facilitate this by providing long-term memory and multi-agent collaboration, respectively. Here’s a step-by-step approach to creating a set of textual instructions or processes for this purpose:

1. Hypothesis Testing
Label: HypothesisTestingProcedure
Instructions:
• Identify the problem or question.
• Formulate a hypothesis that offers a potential solution or explanation.
• Determine the variables and plan an approach to test the hypothesis.
• Collect and analyze data relevant to the hypothesis.
• Interpret the results to either support or refute the hypothesis.
• Conclude based on the evidence and consider alternative explanations.

2. Evidence-Based Reasoning
Label: EvidenceBasedReasoningMethod
Instructions:
• Define the issue or claim to be evaluated.
• Gather all relevant evidence, ensuring a range of perspectives.
• Critically assess the quality and reliability of the evidence.
• Analyze the evidence in relation to the issue or claim.
• Draw conclusions based on the balance and weight of the evidence.
• Reflect on the strength of the conclusions and identify any biases or gaps.

3. Logical Reasoning
Label: LogicalReasoningFramework
Instructions:
• Clearly state the premises or assumptions.
• Use logical operations (e.g., deduction, induction) to derive conclusions.
• Check for logical consistency and coherence in the argument.
• Identify any logical fallacies or errors in reasoning.
• Test the argument against different scenarios or counterexamples.
• Finalize the reasoning process with a logically sound conclusion.

4. Problem-Solving Skills
Label: ProblemSolvingGuideline
Instructions:
• Define and understand the problem.
• Break the problem down into smaller, manageable parts.
• Generate potential solutions or strategies.
• Evaluate each solution's feasibility and impact.
• Select the most effective solution and plan its implementation.
• Execute the plan and monitor its effectiveness, adjusting as necessary.

5. Inference from Limited Information
Label: InferenceDrawingTechnique
Instructions:
• Identify the information available.
• Recognize the gaps in information.
• Use relevant knowledge and principles to infer missing details.
• Consider multiple possibilities and scenarios.
• Evaluate the likelihood of each inference.
• Choose the most probable inferences while remaining open to new information.

6. Making Connections Between Processes
Label: ProcessConnectionStrategy
Instructions:
• List the processes or concepts involved.
• Identify common elements or patterns between them.
• Explore potential interactions and relationships.
• Synthesize a unified understanding or framework.
• Apply this integrative perspective to the task at hand.
• Continuously update as new connections or insights emerge.

Implementation in an LLM Context
• Storage: Store these instructions in the LLM's long-term memory (facilitated by MemGPT).
• Function Calls: Enable function calls to these instructions based on the task or query at hand.
• Combination and Application: Use AutoGen's multi-agent system to dynamically combine different instructions based on the specific problem or goal.
• Adaptation: Allow the LLM to adapt these processes to the context of the problem, drawing on its vast knowledge base and the specific data of the situation.

Considerations
• Flexibility: These instructions should be applied flexibly, adapting to the context and specifics of each task.
• Continuous Learning: While this approach provides a structured framework, it's crucial to remember that current LLMs do not learn or evolve their understanding autonomously post-training.
• Ethical and Practical Limitations: The quality of output will depend on the LLM's training and inherent limitations. Ethical considerations, such as bias and misuse, must be managed carefully.

This framework attempts to mimic some aspects of human problem-solving and learning within the constraints of current AI capabilities. While it can enhance the LLM's performance in specific tasks, it's important to recognize the differences between this simulated problem-solving and genuine human cognitive processes.

[angrysky56]

Not trying to sidetrack the thread because as a non-coder I am lost on autogen, but the main issue is memory, structure and computation cost, I think. If the "brain" needs to be stored in a warehouse and sucks down a bottle of water every image it makes etc that's really no good. Imagine the database contains vast amounts information, books, memories, visuals- how do you store all that, interlink them? How do you process the links efficiently and effectively in real-time? We need a new model framework and hardware. I personally am not a fan of big-tech being the only way to access agi but I am sure they want to keep it that way.

The paper "Efficient shallow learning as an alternative to deep learning" provides a detailed analysis of shallow and deep learning architectures, particularly focusing on their error rates and computational complexities. It demonstrates that shallow networks, like the modified LeNet architecture, can achieve low error rates comparable to those of deep learning architectures but with significantly reduced computational complexity. This finding challenges the prevailing notion that deeper architectures are always superior for complex tasks. The paper also explores the universal behavior of power-law scaling in these architectures and the implications of conservation laws on convolutional layer sizes. It suggests that shallow architectures, inspired by brain-like structures, may offer efficient alternatives for certain machine learning tasks, potentially leading to advancements in hardware development for shallow learning.
https://arxiv.org/ftp/arxiv/papers/2211/2211.11106.pdf