sync : ggml

[ggerganov]

TODO:

• fix examples
• start using backend registry
• update Makefile

[ggerganov]

@slaren I am getting the following assertion after the sync:

make -j && ./main -m models/ggml-base.bin -f samples/jfk.wav

whisper_backend_init: using BLAS backend
whisper_init_state: kv self size  =    6.29 MB
whisper_init_state: kv cross size =   18.87 MB
whisper_init_state: kv pad  size  =    3.15 MB
ggml_backend_sched_alloc_splits: failed to allocate graph, reserving (backend_ids_changed = 1)
ggml_gallocr_reserve_n: reallocating Metal buffer from size 0.00 MiB to 14.01 MiB
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 0.92 MiB
whisper_init_state: compute buffer (conv)   =   17.22 MB
Assertion failed: (src_backend_id != -1), function ggml_backend_sched_split_graph, file ggml-backend.cpp, line 1165.
Process 66825 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000100048950 main`ggml_backend_sched_split_graph(sched=0x000000013890f400, graph=0x0000000148648020) at ggml-backend.cpp:1165:17
   1162	
   1163	               size_t src_id = hash_id(src);
   1164	               const int src_backend_id = sched->hv_tensor_backend_ids[src_id];
-> 1165	               assert(src_backend_id != -1); // all inputs should be assigned by now
   1166	
   1167	               if (src->flags & GGML_TENSOR_FLAG_INPUT && sched->n_copies > 1) {
   1168	                   if (tensor_id_copy(src_id, src_backend_id, 0) == NULL) {

The problem seems to be that the "encode" scheduler does not know about the embd_conv tensor which is the result of the previous "conv" graph. What would be the recommended way to fix this? I think I can copy the data embd_conv data to host memory after the "conv" graph and then copy it back to device memory before calling the "encode" graph. But I wonder if this copy can be avoided.

[slaren]

Should be fixed now, sorry about that.

[ggerganov]

Nice, thank you!

[ggerganov]

@KitaitiMakoto With this PR, the ggml source tree has changed a bit and the Ruby bindings need to be adapted respectively. I'll leave them for now in a broken state, but you can make a PR either to this branch or later to master to resolve the build. Thanks.

[KitaitiMakoto]

Okay, I will make a pull request to master after this pull request will be merged. Thank you for mentioning me.

[ggerganov]

I can't figure out why the whisper.objc CI is failing to use the correct include path. Locally, it runs successfully.