How to enable auto-GPT to pick up where it left off after connection failure? #430

cklose2000 · 2023-04-07T22:04:47Z

Duplicates

I have searched the existing issues

Summary 💡

Any thoughts on enabling auto-GPT to pick up where it left off after an error? It has done some noble work, but has never fully completed the tasks I've set for it before getting cut off by this kind of error:
raise error.APIConnectionError(
openai.error.APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Examples 🌈

No response

Motivation 🔦

No response

SirGunthix · 2023-04-07T22:38:51Z

good Q i dont know yet but am curious. Bumping this thread in hope someone answers

PurrsianMilkman · 2023-04-08T03:33:38Z

i am looking for the same issue. i constantly and i mean constantly get "Error communicating with OpenAI", here is a traceback for more details. this is annoying.

Traceback (most recent call last):
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\urllib3\connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\openai\api_requestor.py", line 516, in request_raw
result = _thread_context.session.request(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\requests\adapters.py", line 547, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\AutoGPT\Auto-GPT\scripts\main.py", line 373, in
memory.add(memory_to_add)
File "C:\AutoGPT\Auto-GPT\scripts\memory.py", line 35, in add
vector = get_ada_embedding(data)
File "C:\AutoGPT\Auto-GPT\scripts\memory.py", line 10, in get_ada_embedding
return openai.Embedding.create(input=[text], model="text-embedding-ada-002")["data"][0]["embedding"]
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\openai\api_resources\embedding.py", line 33, in create
response = super().create(*args, **kwargs)
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
response, _, api_key = requestor.request(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\openai\api_requestor.py", line 216, in request
result = self.request_raw(
File "C:\Users\iliai\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\openai\api_requestor.py", line 528, in request_raw
raise error.APIConnectionError(
openai.error.APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

YaswanthDasamandam · 2023-04-08T17:18:44Z

this is really good feature that we can work on .

marktellez · 2023-04-09T16:58:18Z

it would need a database for starters. i noticed they tried to put in pinecone, but then pinecone blocked it.

i find that the local cache json gets corrupted and then it all goes to hell. I am going to try and fix it with a json parser

i would propose a local database instead of something like pinecone to make it less brittle. sqlite runs on any platform. it could be a possibility.

YaswanthDasamandam · 2023-04-09T17:02:29Z

Finally without pinecode I am stuck with pinecone error if you are working with sqllite I am happy to contribute

BaltekLabs · 2023-04-13T11:38:47Z

I have found a solution to this. You must save the whole interaction in a .txt in the /auto_gpt_workspace. It took some coaxing but I found the command to give it after a restart

Input: I have saved a log of our previous interactions at ./[your_file_name] , read the file and restore yourself to the stop point of the file. You will need to chunk the file in chunks of 6000 tokens.
SYSTEM: Human feedback: I have saved a log of our previous interactions at ./gameDevRetore.txt , read the file and restore yourself to the stop point of the file. You will need to chunk the file in chunks of 6000 tokens.

If it reads the file without the chunking, it goes over the token limit.

This is running on Debian on an EC2 free tier

After this prompt, it loaded the whole file in the buffer and crashed with a max token error.

joeflack4 · 2023-04-24T03:48:30Z

Restarting where it left of feels critical. I'm new to the codebase but if I have time and this isn't fixed this is the first thing I would try to contribute.
@marktellez Where is the local cache JSON stored? I'm only seeing outputs/data related to my agent stored in logs/activity.log.
@cklose2000 I would suggest changing the title to simply "How to enable Auto-GPT to pick up where it left off? ". I think by default we want agents to work this way; that you can end a session for whatever reason and then return where it left off.

ram-sh · 2023-04-29T09:26:32Z

I think by default we want agents to work this way; that you can end a session for whatever reason and then return where it left off.

i couldn't agree more!! i'm trying to solve a complex problem using Auto-GPT and whenever it crash the task starts from the beginning for no good reason.

following @BaltekLabs solution i wanted to add some code to Auto-GPT to do it, first question is if the auto-gpt.json has all the conversation in it or we need to dump the terminal text on each update to a txt file?

ram-sh · 2023-04-29T09:36:16Z

following @BaltekLabs solution i wanted to add some code to Auto-GPT to do it, first question is if the auto-gpt.json has all the conversation in it or we need to dump the terminal text on each update to a txt file?

can someone pls confirm the function below in logs.py is the best place to add an append command to an external text file for each terminal printout?

 def typewriter_log(
        self, title="", title_color="", content="", speak_text=False, level=logging.INFO
    ):
        if speak_text and CFG.speak_mode:
            say_text(f"{title}. {content}")

        if content:
            if isinstance(content, list):
                content = " ".join(content)
        else:
            content = ""

        self.typing_logger.log(
            level, content, extra={"title": title, "color": title_color}
        )

itsDesTV · 2023-05-02T05:18:36Z

I think by default we want agents to work this way; that you can end a session for whatever reason and then return where it left off.

i couldn't agree more!! i'm trying to solve a complex problem using Auto-GPT and whenever it crash the task starts from the beginning for no good reason.

following @BaltekLabs solution i wanted to add some code to Auto-GPT to do it, first question is if the auto-gpt.json has all the conversation in it or we need to dump the terminal text on each update to a txt file?

All you have to do is go to the logs file and preseed the activity file. That will give you a different beginning every time, because if you just restart, then it does the same thing all over. But this will go at the task at a different angle in the beginning. Though giving it the activity log to read in the beginning will 100% get you back to where you want to go. But imo pre seeding will give the AI a better outlook of what to do next. I think that’s better because it’s AI. It’s sometimes dumb but won’t do something it knows isn’t worth it if it already has knowledge of failures it’s done in the past.

joeflack4 · 2023-05-02T05:24:11Z

I agree with @itsDesTV. In theory at least I don't think it sounds complicated. I haven't come across an auto-gpt.json yet, but I think what itsDesTV (and I) are suggesting is that when you start a new session, before making the first call to the LLM (e.g. ChatGPT), we insert the entire prior activity.log history for that agent into the prompt.

itsDesTV · 2023-05-02T05:35:48Z

I agree with @itsDesTV. In theory at least I don't think it sounds complicated. I haven't come across an auto-gpt.json yet, but I think what itsDesTV (and I) are suggesting is that when you start a new session, before making the first call to the LLM (e.g. ChatGPT), we insert the entire prior activity.log history for that agent into the prompt.

Yeah the one thing I havent tried is setting a goal to “read the “activity.txt” (by directing it with the file path) and continue progress from there” and then goal number two would be “read 1.txt” that would have 2 or 3 tasks in it to do, then at the end it would say, once done with prior tasks, “read 2.txt”. I did that a different time but not like this and it works. But you essentially could set a bunch of if/then command in your goals in those files.

For how smart AI is, it honestly pisses me off sometimes lol. I make such good progress and then I try to replicate it and it just never works out. And to do what I truly want to do, it would cost so much :/ bc for Ai to truly be Ai, it needs to fail and learn and fail and learn just like us. That’s why we keep going at it and tweaking all our prompts. But the money it’s spending on us is our time!

joeflack4 · 2023-05-02T05:41:16Z

@itsDesTV Splitting up the agent into multiple tasks sounds a lot like one of the major goals of this issue, and I think some others as well. The thought is a hierarchy of agents, each with its own ai_settings.yaml.

I feel like at some point in the future, some implementation of an LLM will support the ability to get/set a "seed" for a session for replicability. Maybe they will be less powerful, less frequently versioned LLMs; who can say.

itsDesTV · 2023-05-02T05:46:33Z

@joeflack4 so you technically can now if you wanna look into this! I just haven’t dove into it because I simply want to see what I can make one agent do! I’ve already come to terms that only right now, 1 agent, for the most part is useful until it fails. But there HAS to be a way to make it understand. With that being said, multiple separate instances that work together is wild. And if they can make it communicate, that would be EVEN better.

But here is the thread: #3392 (comment)

Boostrix · 2023-05-02T06:11:35Z

With that being said, multiple separate instances that work together is wild. And if they can make it communicate, that would be EVEN better.

only in theory, the complexity is huge - you need to get the sequential approach working first, i.e. where the parent agent literally sleeps/waits for the sub-agent to finish its task, at most monitoring/observing its progress (think a stream/pipe or just a log file - comparing log messages to stated goals/objectives), with the option to "guide" a sub-agent (modify its goals/observe) or to terminate/restart an agent.
The other thing being, progress can be more than a bunch of messages, it may be stuff on disk for instance (folders/files) - so you also need a way to "checkpoint" state of an agent. So that a sub-agent that is restarted may optionally reuse data from a previous invocation/instance.

I don't see any way to make this working and robust without first getting the sequential use-case right. Once you are talking about multiple agents, possibly working in async fashion, the complexity is getting huge - because at that point, you are literally in need of "manager agents" to carefully watch its agent/team of agents, or they will just clobber up your API bills ...

AdinSoftic · 2023-05-02T06:26:39Z

For me it looks like the whole AutoGPT architecture could need management agent that manages the other agents and the whole progress.
That agent could take up where the progress was and feed the memory. It could be used to read the previous sessions.

For the situations where the model gets stuck into "task loop" (because Task prioritization agent doesn't look at the memory and that Execution agent cannot influence the task queue), management agent could overlook the Task creation agent and step up if the task doesn't lead to our goal or if recognizes that he is repeating/getting into a loop.

@Boostrix we came to a similar conclusion. For reducing the bill, that agent could get the summaries instead of all the tiny details. But than again other model needs to make a summary (hard to get those bills down)

Boostrix · 2023-05-02T06:32:37Z

it looks like the whole AutoGPT architecture could need management agent that manages the other agents and the whole progress. That agent could take up where the progress was and feed the memory. It could be used to read the previous sessions.

work on supporting multiple projects and multiple agents to be managed by parent agents is in progress (especially the REST/async efforts will inevitably need a way to assign managers to agents, and possibly teams of agents):

But than again other model needs to make a summary (hard to get those bills down)

right, that is why it would be preferable to "checkpoint" state locally inside that agent's workspace, i.e. using shell scripts or at least /some/ meta info on disk that isn't going through the LLM to help with #3466

eyalk11 · 2023-05-07T02:43:51Z

I missed this post, but I think as a first step, what I suggested #3933 should be implemented. Pickle is the solution, and if there are multiple agents, then it complicates things, but not that much.

Boostrix · 2023-05-09T07:05:25Z

I think we need a draft PR to summarize the main ideas, collated from different issues/talks

EggheadJr · 2023-05-26T13:42:10Z

EggheadJr
/
Solved-How-to-enable-auto-GPT-to-pick-up-where-it-left-off-after-connection-failure-430

joeflack4 · 2023-05-28T04:37:24Z

@EggheadJr Thanks for that, Egghead. This looks kinda like a gist, but in markdown form.
I think implementation may be a little trickier than that when you actually start coding it, but I could be wrong? I stopped working on this because there was some related development going on very quickly, with a lot of different people, and architecture changing. I think they will address this while they address those other things, but I'm not sure.

github-actions · 2023-09-17T01:53:53Z

This issue was closed automatically because it has been stale for 10 days with no activity.

Tymec mentioned this issue Apr 15, 2023

Add memory backend using sqlite and zarr #1457

Closed

5 tasks

Qoyyuum added the enhancement New feature or request label Apr 16, 2023

Boostrix mentioned this issue May 1, 2023

One more long term memory layer #3382

Closed

1 task

Boostrix mentioned this issue May 2, 2023

Not aware of past command+arguments; often enters "forever loops" and repeats the same actions #3644

Closed

1 task

Boostrix mentioned this issue May 3, 2023

Separation of execution and planning into different agents #3593

Closed

1 task

anonhostpi mentioned this issue May 7, 2023

Rearch: Agent State Encapsulation #3717

Closed

Boostrix mentioned this issue May 11, 2023

State management, persistence and resumption #4105

Closed

5 tasks

EggheadJr referenced this issue in EggheadJr/Solved-How-to-enable-auto-GPT-to-pick-up-where-it-left-off-after-connection-failure-430 May 26, 2023

Initial commit

45d12d2

github-actions bot added the Stale label Sep 6, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to enable auto-GPT to pick up where it left off after connection failure? #430

How to enable auto-GPT to pick up where it left off after connection failure? #430

cklose2000 commented Apr 7, 2023

SirGunthix commented Apr 7, 2023

PurrsianMilkman commented Apr 8, 2023

YaswanthDasamandam commented Apr 8, 2023

marktellez commented Apr 9, 2023 •

edited

Loading

YaswanthDasamandam commented Apr 9, 2023

BaltekLabs commented Apr 13, 2023 •

edited

Loading

joeflack4 commented Apr 24, 2023

ram-sh commented Apr 29, 2023

ram-sh commented Apr 29, 2023

itsDesTV commented May 2, 2023

joeflack4 commented May 2, 2023

itsDesTV commented May 2, 2023 •

edited

Loading

joeflack4 commented May 2, 2023

itsDesTV commented May 2, 2023 •

edited

Loading

Boostrix commented May 2, 2023

AdinSoftic commented May 2, 2023 •

edited

Loading

Boostrix commented May 2, 2023 •

edited

Loading

eyalk11 commented May 7, 2023 •

edited

Loading

Boostrix commented May 9, 2023

EggheadJr commented May 26, 2023 •

edited

Loading

joeflack4 commented May 28, 2023

github-actions bot commented Sep 17, 2023

How to enable auto-GPT to pick up where it left off after connection failure? #430

How to enable auto-GPT to pick up where it left off after connection failure? #430

Comments

cklose2000 commented Apr 7, 2023

Duplicates

Summary 💡

Examples 🌈

Motivation 🔦

SirGunthix commented Apr 7, 2023

PurrsianMilkman commented Apr 8, 2023

YaswanthDasamandam commented Apr 8, 2023

marktellez commented Apr 9, 2023 • edited Loading

YaswanthDasamandam commented Apr 9, 2023

BaltekLabs commented Apr 13, 2023 • edited Loading

joeflack4 commented Apr 24, 2023

ram-sh commented Apr 29, 2023

ram-sh commented Apr 29, 2023

itsDesTV commented May 2, 2023

joeflack4 commented May 2, 2023

itsDesTV commented May 2, 2023 • edited Loading

joeflack4 commented May 2, 2023

itsDesTV commented May 2, 2023 • edited Loading

Boostrix commented May 2, 2023

AdinSoftic commented May 2, 2023 • edited Loading

Boostrix commented May 2, 2023 • edited Loading

eyalk11 commented May 7, 2023 • edited Loading

Boostrix commented May 9, 2023

EggheadJr commented May 26, 2023 • edited Loading

joeflack4 commented May 28, 2023

github-actions bot commented Sep 17, 2023

marktellez commented Apr 9, 2023 •

edited

Loading

BaltekLabs commented Apr 13, 2023 •

edited

Loading

itsDesTV commented May 2, 2023 •

edited

Loading

itsDesTV commented May 2, 2023 •

edited

Loading

AdinSoftic commented May 2, 2023 •

edited

Loading

Boostrix commented May 2, 2023 •

edited

Loading

eyalk11 commented May 7, 2023 •

edited

Loading

EggheadJr commented May 26, 2023 •

edited

Loading