Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stable diffusion support with --format image #44

Closed
JasonWeill opened this issue Apr 7, 2023 · 1 comment · Fixed by #66
Closed

Stable diffusion support with --format image #44

JasonWeill opened this issue Apr 7, 2023 · 1 comment · Fixed by #66
Assignees
Labels
enhancement New feature or request @jupyter-ai/magics
Milestone

Comments

@JasonWeill
Copy link
Collaborator

JasonWeill commented Apr 7, 2023

Problem

Users cannot run image models using Stable Diffusion using Jupyter AI, because no existing output accommodates images.

Proposed Solution

Add a --format image option for the %%ai magic command.

If the AI model outputs a URL, -f image should cause the output cell to display the image by remotely loading that URL as an inline image, either using markdown ![alt text](image URL) syntax or HTML <img alt="alt text" src="image URL" /> syntax.

If the AI model outputs binary data, and -f image is specified, the output cell should display the binary data returned. If the HTTP call to the model's API specifies a MIME type in its response headers, that MIME type should be used to decode the binary data.

Update the example notebooks and documentation with information about -f image.

Additional context

The option should apply a prompt template (#42) that should tell the AI model to output an image only, without explanatory text.

@JasonWeill JasonWeill added the enhancement New feature or request label Apr 7, 2023
@JasonWeill JasonWeill added this to the Milestone 2 milestone Apr 7, 2023
@JasonWeill JasonWeill reopened this Apr 12, 2023
@JasonWeill
Copy link
Collaborator Author

Use huggingface_hub:stabilityai/model for Stability AI.

We may need to use an image-compatible HuggingFace provider specifically, not LangChain, for image generation. The current HuggingFace provider only supports text output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request @jupyter-ai/magics
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant