Feat: Bag of words for testing metric

[mallorih]

This PR adds the bag_of_words function to count the frequency of words for evaluation.

Testing

from unstructured.cleaners.core import bag_of_words
string = "The dog loved the cat, but the cat loved the cow."

print(bag_of_words)

[shreyanid]

Could we add a test for text that contains incorrect spaces between words (like how the unstructured output looks sometimes)? for example in "i n t r o d u c t i o n", each letter will be considered its own word right?

[mallorih]

Could we add a test for text that contains incorrect spaces between words (like how the unstructured output looks sometimes)? for example in "i n t r o d u c t i o n", each letter will be considered its own word right?

Are we sure we want to keep all punctuation?

>>> bag_of_words(string)
{'have': 1, 'dog': 1, 'and': 1, 'cat,': 1, 'love': 1, 'my': 1, 'dog.': 1}

[shreyanid]

No, we should not keep all punctuation, only punctuation within a word (hyphens and apostrophes). commas, dashes, between words or end of sentence punctuation should all be removed.

[shreyanid]

Yes, dogs' should (ideally) fall into the same case as dog's because those are both punctuation for only one word, not between words. Could we add a test case for an end of word possessive like dogs' ?

[shreyanid]

looks great! thanks for taking care of all the punctuation cases :)