feat: add calculate edit distance feature

[Klaijan]

Executive Summary

Adds function to calculate edit distance (Levenshtein distance) between two strings. The function can return as: 1. score (similarity = 1 - distance/source_len) 2. distance (raw levenshtein distance)

Technical details

• The weights param is set to default at (2,1,1) for (insertion, deletion, substitution), meaning that we will penalize the insertion we need to add from output (target) in comparison with the source (reference). In other word, the missing extraction will be penalized higher.
• The function takes in 2 strings in an assumption that both string are already clean and concatenated (CCT)

Important Note!
Test case needs to be updated to use CCT once the function is ready. It is now only tested the "functionality" of edit distance, not the edit distance with CCT as its intended to be.

[shreyanid]

1. score (similarity = 1 - percentage) 2. percentage (distance/source_len) 3. distance (raw levenshtein distance)

I'm debating if we need all 3 of these, since the names are getting a little confusing. Should we just stick with score (100% is perfect match, 0 is nothing in common) since that's the accuracy measure that's most easily understood and we'd use for marketing purposes? Not sure when we might want the other values - let me know what use cases you had in mind.

[Klaijan]

@shreyanid I'd say we can keep the score and a raw distance (which would be rename "distance")

[shreyanid]

That sounds more reasonable, ty!

[shreyanid]

LGTM! Since you added a requirement maybe re-pip compile.

Future PRs can revisit the default weights, as well as use actual CCTs for testing.