[Bug]: Vertex LLM does not Handle FunctionCall tools #16678

stfines-clgx · 2024-10-24T19:16:51Z

Bug Description

When using a FunctionTool, I receive the following error:
AttributeError: 'FunctionCall' object has no attribute '_pb'

It seems that this tool makes some assumptions about the input data types.

Version

0.11.7, llama-index-llms-vertex==0.3.7

Steps to Reproduce

Update the code in The multi-agent concierge example to use Vertex, then try and run.

Relevant Logs/Tracbacks

INFO:chromadb.telemetry.product.posthog:Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.
ERROR:asyncio:Exception in callback Dispatcher.span.<locals>.wrapper.<locals>.handle_future_result(span_id='Workflow.run...-6419e73b17c5', bound_args=<BoundArgumen...bose': True})>, instance=<clgx.agent.e...t 0x3230e6a50>, context=<_contextvars...t 0x31fb73e00>)(<WorkflowHand...ibute '_pb'")>) at /Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:273
handle: <Handle Dispatcher.span.<locals>.wrapper.<locals>.handle_future_result(span_id='Workflow.run...-6419e73b17c5', bound_args=<BoundArgumen...bose': True})>, instance=<clgx.agent.e...t 0x3230e6a50>, context=<_contextvars...t 0x31fb73e00>)(<WorkflowHand...ibute '_pb'")>) at /Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:273>
Traceback (most recent call last):
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 283, in handle_future_result
    raise exception
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/CoreSearchAgent.py", line 242, in main
    result = await handler
             ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 290, in __await__
    return self.result()  # May raise too.
           ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 376, in _run_workflow
    raise exception_raised
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 233, in _task
    raise e from None
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 229, in _task
    new_ev = await instrumented_step(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 357, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/events/CoreSearchEvents.py", line 358, in orchestrator
    tool_calls = llm.get_tool_calls_from_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/llms/vertex/base.py", line 498, in get_tool_calls_from_response
    response_dict = MessageToDict(tool_call._pb)
                                  ^^^^^^^^^^^^^
AttributeError: 'FunctionCall' object has no attribute '_pb'
Traceback (most recent call last):
  File "/Users/stfines/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 138, in run
    return loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 243, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/CoreSearchAgent.py", line 242, in main
    result = await handler
             ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 290, in __await__
    return self.result()  # May raise too.
           ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 376, in _run_workflow
    raise exception_raised
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 233, in _task
    raise e from None
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 229, in _task
    new_ev = await instrumented_step(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 357, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/events/CoreSearchEvents.py", line 358, in orchestrator
    tool_calls = llm.get_tool_calls_from_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/llms/vertex/base.py", line 498, in get_tool_calls_from_response
    response_dict = MessageToDict(tool_call._pb)
                                  ^^^^^^^^^^^^^
AttributeError: 'FunctionCall' object has no attribute '_pb'

The text was updated successfully, but these errors were encountered:

dosubot · 2024-10-24T19:19:36Z

Hey @stfines-clgx! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

The error you're encountering, AttributeError: 'FunctionCall' object has no attribute '_pb', suggests that the FunctionCall object is being used in a way that expects it to have a _pb attribute, which it does not. This issue arises in the context of using the get_tool_calls_from_response method in the LlamaIndex framework, specifically with the Vertex LLM integration.

To address this issue, you should check the following:

Verify the FunctionCall Object: Ensure that the FunctionCall object is being correctly instantiated and used. It might be necessary to review the implementation of FunctionCall to understand its attributes and methods.
Check for Updates or Patches: Since this might be a compatibility issue between the versions of LlamaIndex and llama-index-llms-vertex you are using, check if there are any updates or patches available that address this specific problem.
Review the get_tool_calls_from_response Method: This method is designed to extract tool call information from a ChatResponse object. Ensure that the response object being passed to this method is correctly structured and contains the expected data.
Alternative Approaches: If the _pb attribute is expected for serialization or conversion purposes, consider using alternative methods or attributes that are available on the FunctionCall object.

If these steps do not resolve the issue, further investigation into the specific implementation details of the FunctionCall class and its interaction with the Vertex LLM might be necessary.

To continue talking to Dosu, mention @dosu.

logan-markewich · 2024-10-24T19:24:06Z

Probably could be related to the package version for the generative-ai package?

I don't have access to vertex to test, but the code is here

llama_index/llama-index-integrations/llms/llama-index-llms-vertex/llama_index/llms/vertex/base.py

Line 498 in bfd2514

response_dict = MessageToDict(tool_call._pb)

Happy to accept a PR to fix

stfines-clgx · 2024-10-25T00:48:39Z

That does seem possible, esp since the pyproject.toml lists a aiplatform dependency of 1.39.0 and the current version is 1.70.0; I'll pull locally and see if I can build this lib with a newer dependency

logan-markewich · 2024-10-25T15:23:08Z

That would be amazing, thank you!

stfines-clgx · 2024-10-25T16:03:28Z

Taking a look at this, the entire set of interfaces has changed significantly. I've forked and will re-work how to interact with gemini so that it uses the actual released interface and as such doesn't depend on unreleased/private features. which should address the problem.

Once I can verify that it functions I'll submit a PR - it'll probably require a new version update.

stfines-clgx · 2024-11-02T00:00:12Z

Ok this fork has a branch - dev/1_70_protobuf_only that seems to resolve this issue. My testing with several different types of functions and non-function invocations has it working well. It does not resolve
#16625 - it looks like that will take action from the google-cloud-aiplatform team though I will continue to explore mitigations.

@logan-markewich if you would take a look prior to my creating a PR and let me know what else needs doing to make a good PR for this project I would appreciate it.

stfines-clgx added bug Something isn't working triage Issue needs to be triaged/prioritized labels Oct 24, 2024

stfines-clgx linked a pull request Nov 2, 2024 that will close this issue

Update to the latest google-cloud-aiplatform dependency and properly handle function calls in gemini #16793

Draft

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Vertex LLM does not Handle FunctionCall tools #16678

[Bug]: Vertex LLM does not Handle FunctionCall tools #16678

stfines-clgx commented Oct 24, 2024 •

edited

Loading

dosubot bot commented Oct 24, 2024

logan-markewich commented Oct 24, 2024

stfines-clgx commented Oct 25, 2024

logan-markewich commented Oct 25, 2024

stfines-clgx commented Oct 25, 2024

stfines-clgx commented Nov 2, 2024

[Bug]: Vertex LLM does not Handle FunctionCall tools #16678

[Bug]: Vertex LLM does not Handle FunctionCall tools #16678

Comments

stfines-clgx commented Oct 24, 2024 • edited Loading

Bug Description

Version

Steps to Reproduce

Relevant Logs/Tracbacks

dosubot bot commented Oct 24, 2024

logan-markewich commented Oct 24, 2024

stfines-clgx commented Oct 25, 2024

logan-markewich commented Oct 25, 2024

stfines-clgx commented Oct 25, 2024

stfines-clgx commented Nov 2, 2024

stfines-clgx commented Oct 24, 2024 •

edited

Loading