Version 3.0.0 can produce wrong results

[paul-krug]

Hi,

when I use jiwer version 3.0.0 with the following commands:

wer_transform = jiwer.Compose(
    [
        jiwer.ToLowerCase(),
        jiwer.RemoveMultipleSpaces(),
        jiwer.Strip(),
        jiwer.ReduceToListOfListOfWords(),
    ]
    )
wer = jiwer.wer(
        truth = 'This is a short Sentence with a few Words with upper and Lower cases',
        hypothesis = 'His is a short Sentence with a few Words with upper and Lower cases',
        truth_transform = wer_transform,
        hypothesis_transform = wer_transform,
        )

I get wer=0.0714, which is the expected result based on the WER formula.

But when I use the non-deprecated keywords 'reference' and 'reference_transform' instead of 'truth' and 'truth_transform':

wer = jiwer.wer(
        reference = 'This is a short Sentence with a few Words with upper and Lower cases',
        hypothesis = 'His is a short Sentence with a few Words with upper and Lower cases',
        reference_transform = wer_transform,
        hypothesis_transform = wer_transform,
        )

I get wer=0.2857, which is clearly wrong.
This does not happen if the 'jiwer.ToLowerCase()' transform is removed from the transformation, thus I expect the bug to be related to that.

Cheers,
Paul

[nikvaessen]

Thanks for noticing! Fixed in release 3.0.1.