Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Claude-3 plugin with attachments seems to ignore prompt #20

Closed
mhalle opened this issue Oct 30, 2024 · 5 comments
Closed

Claude-3 plugin with attachments seems to ignore prompt #20

mhalle opened this issue Oct 30, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@mhalle
Copy link

mhalle commented Oct 30, 2024

If llm with the llm-claude-3 plugin version 0.6 and the claude-3-5-sonnet-latest model is used with an attachment, the prompt appears to be ignored. That's not true with, say, the gemini model.

Tested with llm -m claude-3-5-sonnet-latest -a image.png "Ignore the image, describe birds"

The output is a the same description of the image, no matter what the prompt. For the latest gemini pro model, in contrast, the interpretation of the image is dependent on the prompt.

--log shows that the prompt and the attachment are at least being logged.

@simonw simonw added the bug Something isn't working label Nov 4, 2024
@simonw
Copy link
Owner

simonw commented Nov 4, 2024

Wow that's a bad bug! Thanks for catching this.

@simonw
Copy link
Owner

simonw commented Nov 4, 2024

llm -m claude-3.5-sonnet 'write a poem about this' -a https://static.simonwillison.net/static/2024/pelican.jpg

This is a beautiful shot of a Brown Pelican perched on some rocky shoreline, with boats visible in the background. The pelican's distinctive profile and long beak are nicely highlighted, and there's a lovely rim lighting effect around its head and neck that creates an almost glowing appearance. The bird appears to be in its adult plumage, with the characteristic gray-brown coloring. Brown Pelicans are common along coastal areas, and this one seems right at home in what looks like a marina or harbor setting. The image has nice depth of field, keeping the pelican sharp while softly blurring the boats in the background.

simonw added a commit that referenced this issue Nov 4, 2024

Verified

This commit was signed with the committer’s verified signature.
strider Gaël Chamoulaud
@simonw
Copy link
Owner

simonw commented Nov 4, 2024

Manual testing of the fix:

% llm -m claude-3.5-sonnet 'extract text from this' -a https://static.simonwillison.net/static/2024/pelican.jpg
There is no text visible in this image. The image shows a brown pelican perched on rocks near what appears to be a marina or harbor, with a boat visible in the background. The pelican is backlit, creating a glowing effect around its head and neck feathers.
% llm -m claude-3.5-sonnet 'write a poem about this' -a https://static.simonwillison.net/static/2024/pelican.jpg
Here's a poem about this pelican:

"Marina Guardian"

Perched upon the rocky shore,
Gray feathers gleaming in morning light,
The pelican stands, watching o'er
The boats that bob in waters bright.

With regal pose and seasoned grace,
Its silhouette against the bay,
Nature's fisher takes its place,
As sentinel through the day.

That ancient beak, so long and strong,
Tells tales of dives and ocean feast,
While backlit plumes, where sunbeams throng,
Create a halo, beauty released.

At harbor's edge, twixt sea and land,
This coastal king maintains its post,
Among the rocks and shifting sand,
A quiet guardian of the coast.
% llm -c 'translate to spanish'
Here's the Spanish translation of the poem:

"Guardián de la Marina"

Posado en la orilla rocosa,
Plumas grises brillando al amanecer,
El pelícano vigila majestuoso
Los barcos que flotan al aparecer.

Con pose real y gracia curtida,
Su silueta contra la bahía,
Pescador natural toma su guarida,
Como centinela durante el día.

Ese pico antiguo, largo y fuerte,
Cuenta historias de buceos y festín marino,
Mientras plumas a contraluz revierte,
Un halo de belleza en su destino.

Al borde del puerto, entre mar y tierra,
Este rey costero mantiene su lugar,
Entre rocas y arena que encierra,
Un guardián silencioso junto al mar.

@simonw simonw closed this as completed Nov 4, 2024
@mhalle
Copy link
Author

mhalle commented Nov 4, 2024

Thanks for tracking this down. If you can push out a new version I would be grateful :)

@simonw
Copy link
Owner

simonw commented Nov 4, 2024

simonw referenced this issue Nov 4, 2024
simonw added a commit that referenced this issue Nov 17, 2024

Verified

This commit was signed with the committer’s verified signature.
strider Gaël Chamoulaud
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants