Add tests for Whisper static pipeline #1250
Conversation
eshiryae
force-pushed
the
b_whisper_static_tests
branch
from
a1bab59 to
48cb438
Compare
|
With transformers 4.46.3 version encoder model have different dynamic shape for input_features, fix for StaticWhisperPipeline - PR #1293 |
eshiryae
force-pushed
the
b_whisper_static_tests
branch
from
e697a90 to
3bab16c
Compare
| @pytest.mark.parametrize("model_descr", get_whisper_models_list(tiny_only=True)) | ||
| @pytest.mark.parametrize("test_sample", | ||
| [ | ||
| # *get_samples_from_dataset(language="fr", length=2), # 1/2 failed |
There was a problem hiding this comment.
What is failed? Do we have ticket for this?
There was a problem hiding this comment.
For one test (with Spanish language), there's mismatch between expected and actual output (looks like language is not detected correctly)
expected: Habritan aguas poco profundas y lo cosas.
actual_out: Habt ihr da noch was poco perfundes und lohosen?
For one test (with French language), there's an error:
RuntimeError: Check '*roi_end <= *max_dim' failed at src\inference\src\dev\make_tensor.cpp:34
I will create tickets for found fails.
There was a problem hiding this comment.
The same issue as in ticket with difference between NPU and CPU outputs.
With fix #1469 - not reproduced.
eshiryae
force-pushed
the
b_whisper_static_tests
branch
2 times, most recently
from
2fdac12 to
0e11d54
Compare
| "test_sample", get_samples_from_dataset(language="de", length=3) | ||
| ) | ||
| @pytest.mark.precommit | ||
| def test_static_whisper_language_de(model_descr, test_sample): |
There was a problem hiding this comment.
What does it actually check? How it's different from test_static_whisper_autodetect?
There was a problem hiding this comment.
Here we explicitly set language in config, in test_static_whisper_autodetect at first additional infer request will be called to detect language of the audio.
| "test_sample", get_samples_from_dataset(language="fr", length=3) | ||
| ) | ||
| @pytest.mark.precommit | ||
| def test_static_whisper_language_fr(model_descr, test_sample): |
There was a problem hiding this comment.
Same question, how it's different from test_static_whisper_autodetect
eshiryae
force-pushed
the
b_whisper_static_tests
branch
2 times, most recently
from
3482704 to
09e1a99
Compare
|
Tested locally, all 17 tests passed with changes from #1469 |
Agree, tests code is similar. But decided to separate them as they cover different cases/functionality of static whisper pipeline. |
No description provided.