Question about UQFF

[schnapper79]

I used your example to create a UQFF file from Lama-3.2-90B.

Now i am really struggling with loading the model from this UQFF-file. i try to use --from-uqff, but without adding -m and a model, it wont run. But if a add -m model it loads this model and safetensor files. So what's the trick?

[EricLBuehler]

Hi @schnapper79! Yes, you need the -a and -m to load this model. Could you please paste the command if there is still an issue?

[gfxenjoyer]

Trying to run mistralrs-server on Windows with

cargo run --features cuda -- -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF --from-uqff llam3.2-vision-instruct-q4k.uqff

results in Error: Expected file with extension one of .safetensors, .pth, .pt, .bin. I even tried adding -a vllama and still got an error.

[gfxenjoyer]

Ignore the comment above. I was able to get it running with the latest commits. I did notice that residual.safetensors was downloaded before the uqff model.

[Jonghwan-hong]

@gfxenjoyer I have same issue. How can i solve this issue? Can you share the solution?

[EricLBuehler]

@Jonghwan-hong can you please share the command you ran?