Could it be possible to run Pixtral-12b in the Comfy UI ?

[JorgeR81]

https://huggingface.co/mistral-community/pixtral-12b-240910

More details here: https://www.youtube.com/watch?v=PfzPfB3esG4

---

Is it technically possible to have Comfy UI support ( native or via custom node ) ?

Could it give better images or have other capabilities beyond Flux ?

[jepjoo]

Yes, similar models like Qwen2-VL *) and MiniCPM already have custom nodes.

I don't think Pixtral is revolutionary in any way, just another late-fusion multimodal model that can interpret images, but output text only.

Early-fusion models that can understand image and text and can also output image and text such as Meta's Chameleon and Transfusion is what's really exciting imo.

*) https://github.com/IuvenisSapiens/ComfyUI_Qwen2-VL-Instruct

[JorgeR81]

From reading the video description, I've got the impression it could also output images ....

Pixtral can create stunningly realistic images from text descriptions, manipulate existing images with precision, and even generate concepts on demand.

[jepjoo]

I think the youtuber is making stuff up. The architecture of the model is basically the same as other late-fusion vision-language models, a text generating LLM as a base (Mistral NeMo 12B in this case) and a vision adapter made to work in tandem with it afterwards.

The early-fusion models are pretty much trained from scratch with image + text dataset.

Some official slides from Mistral (would they really not mention it if it can actually generate images?):

https://x.com/swyx/status/1833932883347865802

Huggingface employee's breakdown on Pixtral:

https://www.reddit.com/r/LocalLLaMA/comments/1fe3x1z/comment/lmkojlp/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[JorgeR81]

Yeah, I also couldn't find other references about generating images.
And the youtuber itself said there was not much information about the model when the video was released ...

[ShmuelRonen]

Yes:

https://github.com/ShmuelRonen/ComfyUI_pixtral_vision

[Vigilence]

Is there a node that runs it locally instead of api?

[ShmuelRonen]

https://github.com/SeanScripts/ComfyUI-PixtralLlamaVision