OpenedAI API Issue

[MinervaArgus]

Describe the bug

Everytime I generate something too long it says this:

raise self.handle_error_response(
openai.error.InvalidRequestError: This model maximum context length is 2048 tokens. However, your messages resulted in over 2054 tokens.

How can I set the limit of the tokens?

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

All i did was make multiple calls to the api using different methods.

Screenshot

No response

Logs

Traceback (most recent call last):
  File "C:\Users\offic\Documents\Python Scripts\open-author\author.py", line 195, in write_chapter
    response = openai.ChatCompletion.create(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 298, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
    self._interpret_response_line(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 765, in _interpret_response_line
    raise self.handle_error_response(
openai.error.InvalidRequestError: This model maximum context length is 2048 tokens. However, your messages resulted in over 2054 tokens.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\offic\Documents\Python Scripts\open-author\author.py", line 312, in <module>
    main()
  File "C:\Users\offic\Documents\Python Scripts\open-author\author.py", line 301, in main
    novel, title, chapters, chapter_titles = write_fantasy_novel(prompt, num_chapters, writing_style)
  File "C:\Users\offic\Documents\Python Scripts\open-author\author.py", line 279, in write_fantasy_novel
    chapter = write_chapter(novel, storyline, chapter_titles[i+1])
  File "C:\Users\offic\Documents\Python Scripts\open-author\author.py", line 205, in write_chapter
    response = openai.ChatCompletion.create(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 298, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 700, in _interpret_response
    self._interpret_response_line(
  File "C:\Users\offic\Documents\Python Scripts\open-author\venv\lib\site-packages\openai\api_requestor.py", line 765, in _interpret_response_line
    raise self.handle_error_response(
openai.error.InvalidRequestError: This model maximum context length is 2048 tokens. However, your messages resulted in over 2054 tokens.

System Info

EVGA NVIDIA RTX 3080 TI

[MinervaArgus]

I meant 2080 TI not 3080

[MinervaArgus]

Update i fixed it by editing the config.yaml and adding truncation_length: length to the model i was trying to use. Now i just want to know why every response gets shorter and shorter everytime almost.

[matatonic]

The truncation length problem is a known issue, see: #3153

[MinervaArgus]

The truncation length problem is a known issue, see: #3153

Yes this was how I fixed it, I was wondering now how come it seems the response gets shorter everytime?

[teddybear082]

@matatonic do you see any light at the end of the tunnel for the truncation length fix? Just curious, not pressing. I'm helping out with something where our main "go to" for local llm generation is textgenwebui's openai extension (thanks so much for making it!)

[matatonic]

@matatonic do you see any light at the end of the tunnel for the truncation length fix? Just curious, not pressing. I'm helping out with something where our main "go to" for local llm generation is textgenwebui's openai extension (thanks so much for making it!)

I do, it will. I had a fix previously (which no longer works). I am currently on holiday and will spend more active time on this near the end of Sept.

[teddybear082]

Thanks!! Really appreciate all your work on this, its brilliant. I've tested a lot of solutions and the textgenwebui + openai extension is by far the easiest to get going. Have a good holiday!

[matatonic]

... it seems the response gets shorter everytime?

Can you explain this in some more detail? Maybe with an example and using OPENEDAI_DEBUG=1? I'm not sure what this could mean.

[MinervaArgus]

... it seems the response gets shorter everytime?

Can you explain this in some more detail? Maybe with an example and using OPENEDAI_DEBUG=1? I'm not sure what this could mean.

That's my bad, it was a problem with my code and how I was iteratively calling it. Now I'm back to the truncation issue lol

[felipemeres]

It took me a bit to figure out that the openai api extension ignored most of the parameters set in the config files that both the webui and the regular api use.

A workaround for the max context problem is to hardcode the value in completions.py for the openai extension as mentioned here:

req_params['truncation_length'] = 8192

I also had to hardcode other model parameters in the completions.py.

[MinervaArgus]

@matatonic is there any way to make the opened ai api publicly available, i.e. to where I could use it in React Native code?

[KartavyaBagga]

Thanks!! Really appreciate all your work on this, its brilliant. I've tested a lot of solutions and the textgenwebui + openai extension is by far the easiest to get going. Have a good holiday!

Did you guys solved the multi user asynchronous feature in the API? if Yes, how ?

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.