Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/amazon nova #292

Merged
merged 8 commits into from
Dec 4, 2024
Merged

Conversation

jean-farmer
Copy link
Contributor

@jean-farmer jean-farmer commented Dec 3, 2024

Description

This PR adds integration for the newly released Nova models in Bedrock. These models support both images and video as inputs.

Image Sample Code

from langchain_aws import ChatBedrock
from langchain_core.messages import HumanMessage

image_path = "media/sunset.png"

llm = ChatBedrock(
    model="us.amazon.nova-lite-v1:0",
    temperature=0.7
)

with open(image_path, "rb") as image_file:
    binary_data = image_file.read()

message = HumanMessage(
    content=[
        {"image": {"format": "png", "source": {"bytes": binary_data}}},
        {"text": "Provide a summary of this photo"},
    ]
)

response = llm.invoke([message])

Video Sample Code

from langchain_aws import ChatBedrock

video_path = "media/cooking-quesadilla.mp4"

llm = ChatBedrock(
    model="us.amazon.nova-lite-v1:0",
    temperature=0.7
)

with open(video_path, "rb") as video_file:
    binary_data = video_file.read()


message = HumanMessage(
    content=[
        {"video": {"format": "mp4", "source": {"bytes": binary_data}}},
        {"type": "text", "text": "Describe the following video"},
    ]
)

response = llm.invoke([message])

Tool Call Sample

from langchain_aws import ChatBedrock
from langchain.tools import tool

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two numbers."""
    return a * b

tools = [multiply]

llm_with_tools = ChatBedrock(
    model="us.amazon.nova-lite-v1:0",
    temperature=1,
    top_p=1,
    model_kwargs={
        "additional_model_request_fields": {
            "inferenceConfig": {
                "topK": 1
            }
        }
    },
).bind_tools(tools)

response = llm_with_tools.invoke([("user", "What is 8*8")])

Note: To get the best results with the Nova models when used in agent, structured or tool calls, use these values for temperature=1, top_p=1 and topK=1 as specified in the samples.


Image and Video used in sample code

sunset
https://github.com/user-attachments/assets/cf803340-18cf-444b-b72b-0e2866c6ce58

@jean-farmer jean-farmer force-pushed the feature/amazon-nova branch 4 times, most recently from 847390a to 0af9549 Compare December 3, 2024 19:21
Copy link
Collaborator

@3coins 3coins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jean-farmer
Thanks for submitting this change. Some minor comments, looks good otherwise.

libs/aws/langchain_aws/chat_models/bedrock.py Outdated Show resolved Hide resolved
libs/aws/langchain_aws/chat_models/bedrock.py Outdated Show resolved Hide resolved
libs/aws/langchain_aws/chat_models/bedrock_converse.py Outdated Show resolved Hide resolved
libs/aws/langchain_aws/chat_models/bedrock_converse.py Outdated Show resolved Hide resolved
samples/models/getting_started_with_nova.ipynb Outdated Show resolved Hide resolved
samples/agents/agents_with_nova.ipynb Outdated Show resolved Hide resolved
libs/aws/langchain_aws/chat_models/bedrock_converse.py Outdated Show resolved Hide resolved
libs/aws/langchain_aws/chat_models/bedrock_converse.py Outdated Show resolved Hide resolved
@jean-farmer jean-farmer force-pushed the feature/amazon-nova branch 3 times, most recently from 3dd3ecd to d2eb6d5 Compare December 3, 2024 22:32
Copy link
Collaborator

@3coins 3coins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jean-farmer
Code looks good 🚀. Checking, if we can add the access to Nova models in the CI account, otherwise will skip the integration tests and merge.

@3coins 3coins merged commit 2ff60dd into langchain-ai:main Dec 4, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants