When collecting information from the web, Auto-GPT visits only the first website from a Google search, then repeats

[LeRobert]

⚠️ Search for existing issues first ⚠️

• I have searched the existing issues, and there is no existing issue for my problem

Which Operating System are you using?

Windows

Which version of Auto-GPT are you using?

Latest Release

GPT-3 or GPT-4?

GPT-4

Steps to reproduce 🕹

I entered the following instructions (not to expect a viable physical model but to observe Auto-GPT actions):

AI's role:
an AI designed to autonomously browse the web and conduct physical research on the topic of ambipolar aether particles

Goal 1: Collect information on ambipolar aether particles from the web, including research from Correa, Aspden, Cope and others.
other goals are not relevant for this issue but they are visible in the prompt section.

When Auto-GPT starts collecting information, it starts with google search and browse_website commands. But it visits only the first website in a list returned from the google command. It overlooks other relevant websites found in the search. Instead of visiting all of them, it conducts a new google search and browses only the first returned website again.

Current behavior 😯

1. Auto-GPT started well, by conducting a Google search on the topic.
   COMMAND = google ARGUMENTS = {'input': 'ambipolar aether particles'}

2. Command google returned 8 websites, some of them relevant.

3. Auto-GPT decided to browse the website of the first search result to gather more information.
   COMMAND = browse_website ARGUMENTS = {'url': 'https://onlinelibrary.wiley.com/doi/full/10.1002/smtd.202000837', 'question': 'ambipolar aether particles'}

4. Command browse_website returned: ('Answer gathered from website: The text does not mention "ambipolar aether particles."

5. Instead of browsing other websites from the first Google search, where the required information is present, Auto-GPT decided to conduct a new Google search to find websites that provide the needed information.

6. This behaviour repeats - upon not finding relevant information in the first website from a list returned by a Google search, it makes a new Google search, thus missing relevant websites already found in previous Google searches.

Expected behavior 🤔

Each Google search returns a list of possibly relevant websites.

Auto-GPT should browse all websites in a list returned from the Google search, not just the first one. After the whole list of websites is browsed and if nothing relevant is found (or not enough of relevant information is found), it should conduct a new Google search.

Your prompt 📝

ai_goals:
- Collect information on ambipolar aether particles from the web, including research
  from Correa, Aspden, Cope and others.
- Prepare a joint physical model describing nature and principles of ambipolar aether
  particles.
- Prepare argumentation how the resulting model extends the standard model of particles.
- Write the resulting model and argumentation to file.
- Shutdown upon achieving your goals.
ai_name: Research-GPT
ai_role: an AI designed to autonomously browse the web and conduct physical research
  on the topic of ambipolar aether particles

Your Logs 📒

No response

[Boostrix]

it might make sense to introduce a dedicated search_online command to fetch search results and save those locally, this could then be parameterized as needed - because you are basically talking about different levels of browsing, i.e. searching first, saving results and then browsing those (possibly locally?)

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[LeRobert]

it might make sense to introduce a dedicated search_online command to fetch search results and save those locally, this could then be parameterized as needed - because you are basically talking about different levels of browsing, i.e. searching first, saving results and then browsing those (possibly locally?)

Yes, this is already implemented to some degree. The issue is that browsing is done only on the first website retrieved from search (which by itself returns a list of many relevant websites). So browsing should continue on other websites in the list before making another search.

[Boostrix]

I think that the chatgpt browser plugin is doing a nice job of iterating over different results, and even doing searches in parallel - we might want to take a look at implementing something similar

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.