[Bug]: google.api_core.exceptions.NotFound: 404 Can not copy from "gs://.../dependencies.tar.gz when deploying a Reasoning Engine agent

[dapolloxp]

File Name

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/reasoning-engine/intro_reasoning_engine.ipynb

What happened?

When deploying a reasoning agent to Vertex AI as a project owner, I get a 404 running the following code block:
remote_agent = reasoning_engines.ReasoningEngine.create(
agent,
requirements=[
"google-cloud-aiplatform==1.62.0",
"cloudpickle==3.0.0",
"pydantic==2.7.4",
"requests",
],
)

My user account is a superadmin and the Reasoning Agent has the proper permissions.

Relevant log output

NotFound: 404 Can not copy from "gs://bucketname/reasoning_engine/dependencies.tar.gz" to "gs://57038048924-140702303983239168-1723816501200/dependencies.tar.gz", check if the source object and target bucket exist. 5: Can not copy from "gs://bucket/reasoning_engine/dependencies.tar.gz" to "gs://57038048924-140702303983239168-1723816501200/dependencies.tar.gz", check if the source object and target bucket exist.

Code of Conduct

• I agree to follow this project's Code of Conduct

[koverholt]

Thanks for reporting this issue and providing details! We've seen similar symptoms in the past when the Reasoning Engine service account is not able to read/write from the GCS staging bucket that you specify, so we typically recommend specifying a new GCS bucket and letting the Reasoning Engine / Vertex AI SDK create the bucket with appropriate permissions, as mentioned in https://cloud.google.com/vertex-ai/generative-ai/docs/reasoning-engine/troubleshooting/deploy#storage-bucket-errors.

However as you mentioned, in this case, it seems like something else is happening in that dependencies.tar.gz might not be getting created or copied into the source bucket. And we're getting similar reports from other users and test failures.

Do you by chance happen to be running this in a newly created bucket or newly created Google Cloud project? Engineering is looking into this failure now, and I'll post updates here as they provide any details on a workaround and/or the fix.

It appears that there is a bug in the Vertex AI / Reasoning Engine SDK, and the engineering team is looking into a fix.

[aniebyl]

Thank you Kristopher for looking into this and recognizing it as a bug in the Vertex AI / Reasoning Engine SDK. I love what you are doing with the Reasoning Engine and can't wait to implement it in the production environment.

[koverholt]

Hello again! The engineering team has pushed a fix that will show up in the next release of the Vertex AI Python client library (google-cloud-aiplatform).

In the meantime, you should be able to downgrade and use a previous version of the Python SDK: pip install google-cloud-aiplatform==1.60.0. And the new version of google-cloud-aiplatform should get released around the end of this week, so once that's out things will be back to normal again. Thanks again for the bug report!

[aniebyl]

Hi Kristopher, I just tested it and it works with google-cloud-aiplatform==1.60.0 Thank you!

[koverholt]

Great, thanks for confirming! I'll keep this issue open until the new Python SDK is published and I'll post an update then.

[koverholt]

A new version of the Vertex AI Python SDK has been released, which includes the fix mentioned here.

So, the sample notebook should be working as expected again as it will install the latest version of google-cloud-aiplatform, which is 1.63.0.

[aniebyl]

Hi Kristopher, is it possible right now to build a RAG with both Vertex AI Search and Google Search using Reasoning Engine?

[koverholt]

Good question! 😄 I responded to the comment in this blog post with some tips and links to sample notebooks & code:

https://www.googlecloudcommunity.com/gc/Community-Blogs/Building-and-Deploying-AI-Agents-with-LangChain-on-Vertex-AI/bc-p/795710/highlight/true#M484

[koverholt]

Closing per #958 (comment). Thanks again for reporting!