Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google-common[major]: Media manager #5835

Merged
merged 59 commits into from
Sep 11, 2024

Conversation

afirstenberg
Copy link
Contributor

@afirstenberg afirstenberg commented Jun 21, 2024

WORK IN PROGRESS - FEEDBACK REQUESTED

There are still other tests and classes to implement, but this should give a first idea of where I see MediaManager as going.

The goal is to provide infrastructure that supports the fileData part in the Gemini API.

  • For AI Studio, this must be a URL generated by the File API, while on Vertex AI, this must be a Google Cloud Storage gs: URI.
  • The idea is that developers will be able to provide an arbitrary URI (ie - an https URL) or data and that this will get mapped to an appropriate supported URI for use in the Gemini API
  • This is useful for image support, and necessary for video and audio support
  • MediaManager is platform (and LLM) neutral, but some of the BlobStore implementations obviously aren't.
  • The Gemini portions of the library will be passed a MediaManager - MediaMager will, itself, have no knowledge of the LLMs

TODO:

  • A BlobStore for the AI Studio File API that backs the data somewhere
  • examples, and documentation

Open questions:

  • How do we get MediaBlob serializable

Will address #5075 #5076

cc: @jacoblee93 @bracesproul @eyurtsev

Copy link

vercel bot commented Jun 21, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchainjs-docs ✅ Ready (Inspect) Visit Preview Sep 11, 2024 4:22am
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
langchainjs-api-refs ⬜️ Ignored (Inspect) Visit Preview Sep 11, 2024 4:22am

@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features labels Jun 21, 2024
@afirstenberg afirstenberg marked this pull request as draft June 21, 2024 02:01
@@ -80,3 +81,49 @@ test("zodToGeminiParameters removes additional properties from arrays", () => {
expect((arrayItemsSchema as any).additionalProperties).toBeUndefined();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey team, just wanted to flag that the new test suite "MediaBlob and BlobStore" includes a test case making an external HTTP request using the fetch method. This change is flagged for review to ensure it aligns with our project's guidelines. Thanks!

@@ -0,0 +1,428 @@
import { v4 as uuidv4 } from "uuid";
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey team, just wanted to flag that this PR introduces a new HTTP request using fetch in the SimpleWebBlobStore class. This comment is just to bring this change to your attention for review. Let me know if you have any questions or need further clarification.

Change default action if invalid blob.
Add "ignore" action for invalid blobs.
Bug fix for BlobStore.store with how it handles the key
@dosubot dosubot bot added the lgtm PRs that are ready to be merged as-is label Aug 23, 2024
Copy link

vercel bot commented Sep 11, 2024

@afirstenberg is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

@jacoblee93 jacoblee93 merged commit 08092cd into langchain-ai:main Sep 11, 2024
37 checks passed
CarterMorris pushed a commit to CarterMorris/langchainjs that referenced this pull request Nov 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features lgtm PRs that are ready to be merged as-is size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants