[Bug]: claude 3.5 vision not work

[faye1225]

What happened?

I followed the documentation to deploy litellm locally and imported Claude 3.5. Ordinary conversations are running smoothly, but the vision feature is not working.

chat：

client = anthropic.Anthropic(api_key='sk-xxxx',base_url='http://127.0.0.1:4000')

message = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
            ],
        }
    ],
)

'''
output: 
# Message(id='chatcmpl-eb17fb44-99cc-450c-9c5b-ea13dda4e9bf', content=[TextBlock(text="I apologize, but I don't see #  any image attached to this conversation. Could you please upload an image or provide a link to the image you'd like me to analyze? Once you do that, I'll be happy to describe what I see in the image for you.", type='text')], model='claude-3-5-sonnet-20240620', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=14, output_tokens=59))
'''

vision 1:

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

base64_image = encode_image("./logo.jpg")

client = anthropic.Anthropic(api_key='sk-xxx',base_url='http://127.0.0.1:4000')

message = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64," + base64_image
                    },
                },
            ],
        }
    ],
)

'''
output: 
Message(id='chatcmpl-e5e015d6-cd62-45a7-9982-3dcb7c996bf3', content=[TextBlock(text="I apologize, but I don't see any image attached to your message. Could you please upload an image and then ask me about it? Once an image is provided, I'll be happy to describe what I see in it.", type='text')], model='claude-3-5-sonnet-20240620', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=14, output_tokens=50))
'''

vision 2:

# https://docs.anthropic.com/en/docs/build-with-claude/vision#how-to-use-vision

client = anthropic.Anthropic(api_key='sk-xxx',base_url='http://127.0.0.1:4000')

message = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                  {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": image1_media_type,
                        "data": image1_data,
                    },
                },
                {
                    "type": "text",
                    "text": "Describe this image."
                }
            ],
        }
    ],
)
'''
output: 
BadRequestError
Error code: 400 - {'error': {'message': "litellm.BadRequestError: AnthropicError - Client error '400 Bad Request' for url 'https://api.anthropic.com/v1/messages'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400\nReceived Model Group=claude-3-5-sonnet-20240620\nAvailable Model Group Fallbacks=None", 'type': None, 'param': None, 'code': '400'}
'''

Relevant log output

No response

Twitter / LinkedIn details

No response