Refactoring CLI to use config file, connect to Llama Index data sources, and allow for multiple agents

[sarahwooders]

Builds on top of #146

Example running on hugging face dataset (loaded with Llama Index):

Updated CLI

Starting Agent

> memgpt run [--agent <agent-name>] 

Configuring defaults

> memgpt configure 

? Do you want to enable MemGPT with Open AI? Yes
? Use OpenAI as default provider? Yes
? Do you want to enable MemGPT with Azure? No
? Select default model (recommended: gpt-4): gpt-4
['/Users/sarahwooders/repos/MemGPT/memgpt']
['sam_pov', 'memgpt_starter', 'memgpt_doc', 'sam_simple_pov_gpt35', 'sam', 'memgpt_doc', 'basic']
? Select default persona: sam_pov
['cs_phd', 'basic', 'basic']
? Select default human: cs_phd
Saving config to /Users/sarahwooders/.memgpt/config

Loading External Data

External data connectors via Llama Index connectors implemented in cli/cli_load.py:

memgpt load [directory/database] [ARGS]

This data source can now be attached to an agent by running memgpt run --data-source <source-name>.

Listing Options (Agents, Humans, Personas, Data Sources)

> memgpt list [agents/humans/personas/sources]

> memgpt list humans

+------------+---------------------------------------------------------------------------------------------+
|    Name    |                                             Text                                            |
+------------+---------------------------------------------------------------------------------------------+
| cs_phd.txt | This is what I know so far about the user, I should expand this as I learn more about them. |
|            |                                                                                             |
|            |                                       First name: Chad                                      |
|            |                                         Last name: ?                                        |
|            |                                         Gender: Male                                        |
|            |                                            Age: ?                                           |
|            |                                        Nationality: ?                                       |
|            |                   Occupation: Computer science PhD student at UC Berkeley                   |
|            |      Interests: Formula 1, Sailing, Taste of the Himalayas Restaurant in Berkeley, CSGO     |
| basic.txt  |                                       First name: Chad                                      |
|   basic    |                                       First name: Chad                                      |
+------------+---------------------------------------------------------------------------------------------+

Adding Personas/Humans

memgpt add [persona/human] --name <name> [-f <filepath>] [--text <text>]

Directory Structure

Configuration, credentials, agent state, and persona/human options are all saved in ~/.memgpt.

~/.memgpt
    /agents 
       /<agent_name> 
          config.json # agent config 
          /agent_state # message state
          /persistence_manager # persistence manager state
      ...
    /humans
        <human_name> # file with human persona 
        ...
    /personas
        <persona_name> # file with persona  
        ...
   /archival 
       /<index-name> # directory containing saved Llama Index 
        ...
   config # config file 

Legacy Support

Both legacy commands should still work:

> memgpt 
> python main.py

To-Dos

• Connect Llama Index embedding configuration with MemGPTConfig values
• Graceful Ctrl-D/Ctrl-C exits
• Test agent checkpointing
• Beautify messages ✨

[cpacker]

If you say no to using OpenAI, and let the process crash out:

Then you run memgpt run afterwards and say "No" to select existing agent, it ends with:

instead of taking back to the first loop asking for an OpenAI key / if you want to use OpenAI (if you set OPENAI_API_KEY it allows you to progress)

[sarahwooders]

@cpacker do you have a local model endpoint to test with? I fixed the hugging face embeddings model, so the vector index should work locally now.