fix: fix regression in OctoAI llm provider after 0.11

[masci]

Description

With the release of 0.11 I broke the OctoAI llm while refacoring the code to remove ServiceContext. This PR fixes the regression.

Fixes comment by @mbonet

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

[mbonet]

@masci sorry for not pushing the PR on time, something else broke.
The proposed changes are not sufficient to make it work it still fails:

/site-packages/llama_index/llms/octoai/base.py", line 147, in chat
    response = self._client.chat.completions.create(
AttributeError: 'OctoAI' object has no attribute 'chat'

[masci]

@mbonet no worries and thanks for the review! Unfortunately this integration has no tests, so I'm blindly updating the code as I see fit, I'll figure out how to add a basic e2e test, in the meantime if you can keep testing my branch that would be invaluable for me :)

[mbonet]

@masci
No problem!

The last version is throwing some errors:

Traceback (most recent call last):
  File "app/anto.py", line 47, in <module>
    octoai = OctoAI(model="meta-llama-3.1-405b-instruct", token=os.getenv("OCTOAI_API_KEY"), max_tokens=130)
  File "/home/user/project/venv/lib/python3.8/site-packages/llama_index/llms/octoai/base.py", line 88, in __init__
    super().__init__(
  File "/home/user/project/venv/lib/python3.8/site-packages/pydantic/main.py", line 193, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for OctoAI
max_tokens
  Field required [type=missing, input_value={'callback_manager': <lla..., 'output_parser': None}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.8/v/missing

It's complaining about max_tokens and this is the instantiation I'm doing:

octoai = OctoAI(model="meta-llama-3.1-405b-instruct", token=os.getenv("OCTOAI_API_KEY"), max_tokens=130000)

Also tried without max_tokens arg, same result

[nerdai]

Looks good. Just wondering did octoai update?

• we use api_key instead of token for constructing Client (or OctoAI)
• we use OctoAI.text_gen.create_chat_completion instead of OctoAI.chat.completions

[masci]

@nerdai it looks like the SDK changed but hard to tell how and when, the python code is generated with fern and the repo is private it seems, release notes are not helpful. We pin the client with ^1.6.0 so we're currently depending on 1.10 on fresh installs.

[masci]

@mbonet I should've fixed the pydantic errors, OctoAI class intantiates fine on my branch, can you give it a shot?

[mbonet]

@masci good job!
Now we don't have compilation errors but we have API errors.

If we initiate octoai - without max-tokens, it's not complaining as mandatory value, and it might potentially be.

This is the error

octoai.core.api_error.ApiError: status_code: 400, body: {'object': 'error', 'message': "Invalid request: {'body.messages.0.content': ['Field required'], 'body.messages.1.content': ['Field required'], 'body.max_tokens': ['Input should be a valid integer']}", 'type': 'invalid_request_error', 'param': None, 'code': None, 'validation_errors': {'body.messages.0.content': ['Field required'], 'body.messages.1.content': ['Field required'], 'body.max_tokens': ['Input should be a valid integer']}}

When we instantiate as:

octoai = OctoAI(model="meta-llama-3.1-405b-instruct", token=os.getenv("OCTOAI_API_KEY"))

Despite this controversial point, we have a bigger issues which is the following, when we pass the max_tokens like

octoai = OctoAI(model="meta-llama-3.1-405b-instruct", token=os.getenv("OCTOAI_API_KEY"), temperature=0.1, max_tokens=130000)

We get this error:

octoai.core.api_error.ApiError: status_code: 400, body: {'object': 'error', 'message': "Invalid request: {'body.messages.0.content': ['Field required'], 'body.messages.1.content': ['Field required']}", 'type': 'invalid_request_error', 'param': None, 'code': None, 'validation_errors': {'body.messages.0.content': ['Field required'], 'body.messages.1.content': ['Field required']}}

So the content we relay to the API is not compliant with octoai as it was before :/

[masci]

@mbonet I managed to get an OctoAI account and I could test the actual calls. I think I fixed both the problems you reported that I was able to reproduce, let me know if this new version works for you!

[mbonet]

@masci excellent job!

I just encountered this issue:

octoai.core.api_error.ApiError: status_code: 400, body: {'object': 'error', 'message': "Invalid request: {'content': 'Either content or tool_calls must be provided.'}", 'type': 'invalid_request_error', 'param': None, 'code': None, 'validation_errors': {'content': 'Either content or tool_calls must be provided.'}}

For sure I have pulled your latest changes:

And how I invoke it is as it was before

octoai.chat([ChatMessage(role="user", content=f"""Hello""")])

[masci]

@mbonet I think the latest changes to LlamaIndex would imply your call should be rewritten like this:

from llama_index.core.base.llms.types import ChatMessage, MessageRole


octoai.chat([ChatMessage(role=MessageRole.USER, content="Who is Paul Graham?")])

the above works for me, I also added it as a test case.

[mbonet]

@masci sorry you are right, I had a cache version, my bad!
Great job!

👍