[Roadmap]: Complex Tasks Work Items (GAIA)

[afourney]

General tasks, work items, and issues, related to solving complex tasks, as defined by the GAIA benchmark, and executed by AutoGenBench (#973).

GAIA Refinement (things to fix from the March 1st run)

Beta Give feedback

1. Check if files are too large before putting them into the initial question prompt. If too large, instruct to open with web_surfer using a file:/// URI.
2. Prevent or greatly discourage parallel tool use in WebSurfer. Through prompting or code, ensure only one function is called at a time.
3. Ensure that, when code is written by Assistant, it gets a chance to run in Computer_Terminal (hardcode this!, it has been stubbornly hard to fix with prompting)
4. Ensure when Computer_Terminal throws a ModuleNotFoundError, the assistant responds with instructions to do a pip install ((hardcode this!, it has been stubbornly hard to fix with prompting)
5. Update Computer_Terminal's default response to indicate that the proper way to address it is by providing it code to run in markdown code blocks.
6. Create a 50 or 60-question subset of GAIA, across levels, for rapid iteration (e.g., 20, 20, 10, levels 1, 2, and 3 problems)
7. Complete and merge Allow costs and window size limits to be specified in the config_list #1682
8. Navigational search doesn't work when links have parentheses in them. Fix regex here: https://github.com/microsoft/autogen/blob/63a01753114696ed51f9f7740d0f0fcc33c0c8b8/autogen/agentchat/contrib/web_surfer.py#L157-L159
9. Filter data URIs from Markdown

Immediate Priority

Beta Give feedback

1. Update SocietyOfMind template to use the latest AutoGen features like initiate_chats and iPython executors
2. Develop a purpose-built planning and reasoning module to oversee progress on tasks #1630
   gaia +1
3. Develop a shared short-term memory , or ledger, separate from conversation history. #1671
   +0

High Priority

Beta Give feedback

1. Develop a version of WebSurferAgent that uses a proper headless browser (perhaps with multimodal LLM support) #1481
   gaia +1
2. Add a specialized Agent, similar to WebSurfer, for browsing and operating on local files. #1765
   gaia +1
   

Longer-Term Priority

Beta Give feedback

1. Re-add "find-in-page" functionality to WebSurferAgent
2. Explore methods for measuring progress when agents are running (e.g., to decide when to break loops, change plans, terminate, etc.) #1670
   gaia +1
3. Develop a (possibly static, possibly internal) dashboard for tracking progress.
4. Integrate with AgentEval -- track progress on soft-metrics. #1550
   research roadmap +2
5. Evaluate and Track SLMS (7B, 13B) Models and Report Results
6. Support Workflows from AutoGen Studio in AutoGen Bench (Users create and tweak their agent workflows and can automatically run it on autogenbench)

Other Tasks

Beta Give feedback

1. Onboard other complex task benchmarks like webarena, miniWoB++, Mind2Web

Completed

Click to expand

Completed tasks

Beta Give feedback

1. Create a script for collating GAIA runs into the JSON format needed to submit to the leaderboard.
2. Create a shared folder, and establish a convention, for sharing latest run logs etc.
3. Create a protected branch of AutoGen (e.g., complex_tasks) for coordinating experiential work on complex tasks.
4. Greatly enhance telemetry and logging to cover timestamp events, and track costs #1368
   +0
   
5. Update the society_of_mind_gaia branch (used by the benchmark) to the latest version of AutoGen #1477
   gaia +1
   
6. Extend GAIA reporting to include measures of cost, task completion time, number of messages, etc. #1488
   gaia +1
   
7. Update GAIA agents to use gpt-4-1106-preview and report results #1489
   gaia +1
8. Produce a full set of validation benchmark numbers. Not just level 1.
9. Add a "file parser" that can handle converting a broad range of files to text #1562
   gaia +1
   

   
10. Add the ability to access YouTube transcripts and video descriptions to WebSurfer #1563
    gaia +1
    
11. Add multi-modal support to GAIA agents -- they should be able to handle images. Audio and video are a lower priority. #1549
    gaia +1
12. Explore methods to handle long-running contexts. This is especially important with level 2 and 3 problems. #1478
    +0
    
13. Experiment with alternate GroupChat prompts (like GroupChat moderator, or perhaps graph-based GroupChat)
14. Get WebSurfer to make more use of answer_from_page or summarize_page rather than repeated calls to page_down
15. Get the main GAIA agent to be more willing to output an educated "guess" of the final answer rather than just saying "unable to determine"

[afourney]

When items are converted into issues, it means that work is in progress.