Improve timestamp accuracy

[achimmihca]

Thank you for porting Whisper to Unity!

One issue I face is that the timestamps are not very accurate.
I found that there are related projects that try to improve this

• https://github.com/m-bain/whisperX
• https://github.com/jianfch/stable-ts
• https://github.com/linto-ai/whisper-timestamped

Is it possible somehow to make use of these to improve timestamp accuracy?
What would be necessary to achieve this?

Note that there are also related issues on whisper.cpp

• Implement word-level timestamps approach proposed by OpenAI ggerganov/whisper.cpp#375

[Macoron]

Have you looked into DTW? ggerganov/whisper.cpp#1485

I believe there is a new setting option here, but I didn't test them yet:

whisper.unity/Packages/com.whisper.unity/Runtime/Native/WhisperNativeParams.cs

Lines 95 to 102 in 3fa68e0

	// [EXPERIMENTAL] Token-level timestamps with DTW
	[MarshalAs(UnmanagedType.U1)] bool dtw_token_timestamps;
	WhisperAlignmentHeadsPreset dtw_aheads_preset;
	int dtw_n_top;
	WhisperNativeAheads dtw_aheads;
	UIntPtr dtw_mem_size; // TODO: remove

[achimmihca]

Have you looked into DTW?

Thanks for pointing this out. But I don't see an C# API yet to try this out.

From the comment in WhisperNativeParams.cs I think it is only a point in time ("moment in audio") but I would need start and end for each word

        // [EXPERIMENTAL] Token-level timestamps with DTW
        // do not use if you haven't computed token-level timestamps with dtw
        // Roughly corresponds to the moment in audio in which the token was output
        ulong t_dtw;

Anyway, will give it a try when it is ready.