[Feature] Conversation Summary Mode: Track Assistant Responses #868
Conversation
- Added support for new conversation summary mode - Changes made to six files: ai_config.py, config.py, data.py, json_parser.py, main.py, and summary_tests.py - Toggle on or off with '-cs' or '--conversation-summary' args when running main.py - Added step summarization, which prints to console and appends to file a summarization of each step - Added final summarization, which condenses all summarized steps into an even shorter summary (via a GPT agent) then prints it to console and appends it to file upon exit ✨ feat(main.py): refactor interaction loop and add support for conversation summary mode - Refactored interaction loop to include conversation summary mode support - Print the step summary and append it to the summary file if conversation summary mode is enabled - Print the final summary and append it to the summary file if conversation summary mode is enabled upon program exit ✨ feat(conversation_summary_mode): add Summary and SummaryUtils classes, implement text chunk splitting, and add tests - Added Summary and SummaryUtils classes to handle conversation summary mode - Created summary object for printing step summaries to console and appending to file - SummaryUtils class includes utility methods for splitting sentences into bullet points and updating JSON schema and prompt - Implemented append_final_summary and split_text_into_chunks methods for handling large text by splitting it into smaller chunks - Added tests for all methods ✨ feat(config.py): add conversation_summary_mode and related methods - The model variable in the agents dictionary has been renamed to gpt_agent_model to improve clarity - The Config class now has a conversation_summary_mode attribute, which is set to False by default - Two prompts have been added to the Config class to be used in conversation summary mode - The set_conversation_summary_mode method has also been added to the Config class to allow for the conversation_summary_mode attribute to be set - The step_summarization_prompt and final_summarization_prompt can be set inside .env file ✨ feat(data.py): add support for conversation summary mode - The load_prompt function now checks if the conversation_summary_mode flag is set in the Config object - If it is, the function adds the summary field to the response structure in the AutoGPT prompt using the SummaryUtils.add_summary_field_to_prompt method - This allows the Summary class to extract the step summary from the assistant's reply and log it to the summary file ✨ feat(json_parser.py): add support for conversation summary mode by calling SummaryUtils.add_summary_field_to_json() to add a summary field to the JSON schema ✨ feat(main.py): initialize the Summary object if conversation summary mode is enabled; Print the step summary and append it to the summary file if conversation summary mode is enabled - If conversation summary mode is enabled, the JSON schema is modified to include a summary field 🚀 feat(summary_tests.py): add unit tests for Summary and SummaryUtils classes - This commit adds unit tests for the Summary and SummaryUtils classes - The tests cover the initialization of a Summary instance, the incrementation of the step counter, the printing of the step and final summaries to the console, the appending of the step summary and result to the summary file, the formatting of the final summary prompt, and the extraction of the step summary from the assistant's reply refactor(.env.template): add new env variable examples to .env.template - Show examples of how to set STEP_SUMMARIZATION_PROMPT and FINAL_SUMMARIZATION_PROMPT inside .env file
scriptogre
changed the title
|
@scriptogre Thank you so much for tackling exactly what I was looking for. In the current state, I find the meta-level action strategy alone fascinating - to capture it in an automated descriptive way is extremely nice. It is extremely exciting to put Auto-GPT on a question where it works as a researcher. If you know your stuff and have put hundreds if not thousands of hours into it, it's super exciting to see how Auto-GPT eventually (when it's not busting its balls over a JSON error or losing its memory again) tends to act similarly, so you can say that 95% of its actions are really useful and interesting - and 1000x faster. The topic of reporting/extraction of knowledge and automatic knowledge discovery and summarization/structuring I feel has huge potential and is underrated. PS: Very off-topic, but a "secret desire" that would be pretty nice, if it could be fed (embeddings?) with available information, it starts to validate and extend. It doesn't need to fail trying to download scientific papers, if there is a pretty nice Zotero library with structured and rich content available. Maybe also interconnecting it with Galactica... That would be my ultimate dream. |
|
Wow, feels good to know my work proves to be of use to other users. My intention is to develop a succinct summarization that can be fed back to the AI model, ensuring it remains focused and eliminating the need to reset it after each terminal session. By maintaining a record of past actions and objectives, the AI can seamlessly resume its tasks. The ultimate goal is to generate an efficient, multi-layered summary that encapsulates the AI's progress and ongoing tasks, effectively communicating its status in a clear and concise manner. |
…summary counter incrementation This commit addresses two issues in main.py: the final summary not appending to the output file upon exit and the step summary counter not incrementing as expected. These problems occurred due to missing code pieces after resolving merge conflicts.
scriptogre
changed the title
scriptogre
changed the title
|
@Torantulino What do you think? Any issues with the PR? It's a pretty nice functionality for making the auto-gpt a more robust tool. I hope you consider merging it |
|
@scriptogre There are conflicts now |
Yup! Seen those. Was about to fix them today but didn't get the chance. Will fix them tomorrow. |
github-actions
bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
2 similar comments
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
This is a mass message from the AutoGPT core team. For more details (and for infor on joining our Discord), please refer to: |
|
This would also seem like something that would be best done using dynamic prompting as per #3937 |
|
Interesting concept! It seems somewhat similar to the rolling summary that is currently being used for the agent's message history. If anyone is interested in re-evaluating the value of this PR and comparing it to the current implementation of MessageHistory that would be grand. For now, this PR has extensive merge conflicts so I'll close, with the hope that this will be looked into further. |
Pwuts
added
the
fridge
Background
This pull request introduces the "Conversation Summary Mode," which generates concise GPT-generated summaries for each step and a final summary upon exit. It enables users to track the assistant's responses and identify deviations, ultimately improving its functionality.
To activate the mode, run main.py with -cs or --conversation-summary:
Changes
Introduced Conversation Summary Mode, generating concise step summaries and a final summary upon exit
Refactored main.py to support the new mode while maintaining existing functionality
Created Summary and SummaryUtils classes to manage summarization features and handle large text
Added test coverage for all new methods and classes
Updated .env file to accept STEP_SUMMARIZATION_PROMPT and FINAL_SUMMARIZATION_PROMPT variables, allowing users to customize prompts and improve summarization results
Documentation
The new features are accompanied by comprehensive docstrings for each newly added method, code comments where needed, and an updated README.md section explaining how to set up and use this mode. All changes are modular, and the existing code has been minimally altered to adhere to the Separation of Concerns principle.
Test Plan
Tests cover the initialization of a Summary instance, incrementation of the step counter, printing of step and final summaries to the console, appending of step summaries and results to the summary file, formatting of the final summary prompt, and extraction of the step summary from the assistant's reply.
PR Quality Checklist
[?] My pull request is atomic and focuses on a single change
It's not really a single change, but it's a very modular change. I tried my best not to touch the existing code too much but keep all the new functionality inside the conversation_summary library for applying the Separation of Concern principle.
[✓] I have thoroughly tested my changes with multiple different prompts.
[✓] I have considered potential risks and mitigations for my changes.
[✓] I have documented my changes clearly and comprehensively.
[✓] I have not snuck in any "extra" small tweaks changes
Potential risks and mitigations:
Risk: Summaries may not always be coherent or helpful due to GPT's inherent limitations.
Mitigation: Carefully monitor the output to ensure the feature is working as expected and adjust the prompts and code as needed to improve the quality of summaries.
Risk: Large texts or numerous steps may cause the script to take a long time to generate summaries or hit API rate limits.
Mitigation: Implement chunking to handle large texts and limit the number of steps during testing. Additionally, monitor API usage to avoid unexpected costs or throttling.
Risk: Unexpected bugs or issues may arise when the new mode is enabled or when working with the modified code.
Mitigation: Thoroughly test the changes and monitor the output during usage. Address any issues as they arise and consider adding more tests to cover edge cases.