gguf : add 64-bit support (GGUF v2)

[ggerganov]

Adding 64-bit support as discussed: ggerganov/ggml#302 (comment)

Help with testing is appreciated. Should be backward compatible with v1

[klosax]

We should add types uint64_t , int64_t and double
And we should change to uint64_t on all lengths / sizes / counts just to be safe and future-proof, not only for tensor dimensions.

[ggerganov]

Need some help with the Python code

In the meantime, I will now add V1 backward comp in ggml.c reading

[klosax]

We should change to uint64_t on all lengths / sizes / counts just to be safe and future-proof, not only change tensor dimensions.

[KerfuffleV2]

I tested loading a couple GGUF v1 models, the backward compatibility seems to work fine.

[ghost]

Similarly, no issues loading various v1 models.

[klosax]

Both versions work good.

[klosax]

We can actually use quantizeto losslessly convert gguf v1 to v2, if the same format is chosen.

[philpax]

Looks good, is the plan to update the metadata values for the lengths/etc before merge?

[ghost]

We can actually use quantizeto losslessly convert gguf v1 to v2, if the same format is chosen.

@klosax Ah, that's useful. For a 7b q4_0 model, I use ./quantize ~/wizardLM.gguf 2 3

I don't need --allow-requantize or --leave-output-tensor, right?

[klosax]

I don't need --allow-requantize or --leave-output-tensor, right?

I dont think those parameters are needed. Maybe we should have a new parameter --copy-all-tensors instead so quant format wont matter.

[KerfuffleV2]

I dont think those parameters are needed.

llama.cpp/llama.cpp

Lines 4743 to 4746 in 730d9c6

	// quantize only 2D tensors
	quantize &= (tensor->n_dims == 2);
	quantize &= params->quantize_output_tensor || name != "output.weight";
	quantize &= quantized_type != tensor->type;

That logic is actually kind of wrong because the k-quants stuff can choose a different type than quantized_type. There's also no check after that part to see if the special k-quants type is the same as the current tensor type, it just tries to quantize (or fails if --allow-requantize isn't set).

It probably will work for the non-k-quants types but pretty sure k-quants won't work. (There were also some changes to the decisions k-quants makes for LLaMA2 70B models so in that particular case it wouldn't pass through all the tensors even if the other issues were dealt with.)

[ghost]

Thanks. I used quantize q4_0 wizardlm and llama2. They load as GGUF V2, and appear working. I'll beware quantize and k-quants.

[ggerganov]

Thanks everyone for testing. We should merge this - anything else we won't to try before this?

[pudepiedj]

I am a long-term enthusiast for whisper.cpp which I use by default nowadays to transcribe my podcast Unmaking Sense.
I am new to Llama, so apologise if this isn't useful, but a few comments:

1. A big thank you for all the work on both these projects, which are exemplary.
2. I've quantized successfully this morning from the original Meta AI download llama-2-13B-chat through F16 to q8_0 GGUF and it runs straight away on a MacBook Pro M2 Max with 32GB RAM. q4_0 also runs, of course, but that isn't new.
3. It says "Ctrl-C" allows interaction, but mine just aborts when running in terminal, rather as I would expect. I am obviously missing something.
4. I am not sufficiently technically competent to offer much by way of coding collaboration but I'd happily write some user documentation if it would help coming from someone starting out on this journey who asks daft questions based on impressive levels of ignorance.
5. Question: is there a way to prevent Llama-2-13B from producing random responses of indeterminate length and almost no relevance? In the screenshots in the repo you seem to have managed to force it to do the "meaning of life" question repeatedly, but I have no idea how you make it do that or control what kind of content it produces.
6. Sorry to waste your time if this isn't helpful or breaches github protocols in some way.

[Green-Sky]

3. It says "Ctrl-C" allows interaction, but mine just aborts when running in terminal, rather as I would expect. I am obviously missing something.

did you press it more than once? It queues a stop and gives you the control, and then if pressed again, exits the program. try to play with it a bit more :)

5. Question: is there a way to prevent Llama-2-13B from producing random responses of indeterminate length and almost no relevance? In the screenshots in the repo you seem to have managed to force it to do the "meaning of life" question repeatedly, but I have no idea how you make it do that or control what kind of content it produces.

did you use the prompt template?

[pudepiedj]

3. It says "Ctrl-C" allows interaction, but mine just aborts when running in terminal, rather as I would expect. I am obviously missing something.

did you press it more than once? It queues a stop and gives you the control, and then if pressed again, exits the program. try to play with it a bit more :)

It seems that if you use Ctrl-C while the assistant is printing a reply, it behaves as expected and described, but if you press it afterwards, it aborts. Thanks for the hint.

5. Question: is there a way to prevent Llama-2-13B from producing random responses of indeterminate length and almost no relevance? In the screenshots in the repo you seem to have managed to force it to do the "meaning of life" question repeatedly, but I have no idea how you make it do that or control what kind of content it produces.

did you use the prompt template?

I hadn't, but now I have. Thank you, again. Unfortunately it seems to lead to a collapse of the quality of the response to a point where it is worthless, but I therefore obviously need to investigate the process more.

[KerfuffleV2]

If you'd need to follow up, I'd suggest making an issue specifically to discuss your problem. This is a pull request that doesn't seem directly related.