Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Error responses are not handled correctly for google openapi/openrouter #527

Open
sambhav opened this issue Dec 22, 2024 · 2 comments · May be fixed by #551
Open

[BUG] Error responses are not handled correctly for google openapi/openrouter #527

sambhav opened this issue Dec 22, 2024 · 2 comments · May be fixed by #551
Labels
refactor change to code structure

Comments

@sambhav
Copy link

sambhav commented Dec 22, 2024 •
edited
Loading

In case the API returns a 429/Rate limit exceeded, pydantic-ai throws a date-time parsing exception instead of surfacing the appropriate error message from the API around RLE(rate-limit-exceeded).

This can easily be replicated by using openrouter with one of the free gemini models.

from pydantic_ai import Agent

from pydantic_ai.models.openai import OpenAIModel

model = OpenAIModel(
    "google/gemini-2.0-flash-exp:free",
    base_url="https://openrouter.ai/api/v1",
    api_key="key",
)

agent = Agent(
    model=model,
    system_prompt='Be concise, reply with one sentence.',  
)

result = agent.run_sync('Who are you?')
print(result.data)

The above returns -

Traceback (most recent call last):
  File "/Users/sam/dev/openai/openai_demo.py", line 32, in <module>
    result = agent.run_sync('Who are you?')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sam/dev/openai/.venv/lib/python3.12/site-packages/pydantic_ai/agent.py", line 327, in run_sync
    return asyncio.get_event_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/sam/dev/openai/.venv/lib/python3.12/site-packages/pydantic_ai/agent.py", line 255, in run
    model_response, request_usage = await agent_model.request(messages, model_settings)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sam/dev/openai/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 152, in request
    return self._process_response(response), _map_usage(response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sam/dev/openai/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 207, in _process_response
    timestamp = datetime.fromtimestamp(response.created, tz=timezone.utc)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object cannot be interpreted as an integer

This happens because the error response is not correctly handled in _process_response -

ChatCompletion(id=None, choices=None, created=None, model=None, object=None, service_tier=None, system_fingerprint=None, usage=None, error={'message': 'Provider returned error', 'code': 429, 'metadata': {'raw': '{\n  "error": {\n    "code": 429,\n    "message": "Quota exceeded for aiplatform.googleapis.com/generate_content_requests_per_minute_per_project_per_base_model with base model: gemini-experimental. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.",\n    "status": "RESOURCE_EXHAUSTED"\n  }\n}\n', 'provider_name': 'Google'}}, user_id='user_...')

We should check for the presence of the error object and handle the other fields appropriately.

Note: I have noticed this with both google's OpenAI compat API and openrouter's gemini API.

This is what an example output response may look like
@sambhav sambhav changed the title [BUG] Error responses are not handled correctly. [BUG] Error responses are not handled correctly Dec 22, 2024
@sambhav sambhav changed the title [BUG] Error responses are not handled correctly [BUG] Error responses are not handled correctly for google openapi/openrouter Dec 22, 2024
@sambhav
Copy link
Author

sambhav commented Dec 22, 2024

This seems to happen due to inappropriate handling of type-casting by the openai client. OpenAI client always casts the response to a ChatCompletion response type, which they allow additional fields to be set on.

https://github.com/openai/openai-python/blob/89d49335a02ac231925e5a514659c93322f29526/src/openai/_models.py#L87-L100

@sydney-runkle
Copy link
Member

PRs welcome! Thanks for outlining the issue clearly.

@sydney-runkle sydney-runkle added the refactor change to code structure label Dec 23, 2024
@sambhav sambhav linked a pull request Dec 26, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
refactor change to code structure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Validate chatcompletion to avoid unexpected bugs sambhav/pydantic-ai
2 participants
@sambhav @sydney-runkle