feat: add RecursiveSplitter component for Document preprocessing

[davidsbatista]

Related Issues

• fixes Add a Recursive Chunking strategy #8548

Proposed Changes:

• Adding a RecursiveSplitter, using a set of predefined separators to split text recursively - see issue for more details

How did you test it?

• local unit tests and integration tests plus CI tests

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 12357154921

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage increased (+0.1%) to 90.601%

---

Files with Coverage Reduction	New Missed Lines	%
components/preprocessors/sentence_tokenizer.py	1	94.12%

Totals
Change from base Build 12353137414:	0.1%
Covered Lines:	8222
Relevant Lines:	9075

---

💛 - Coveralls

[davidsbatista]

@bglearning - mentioning you since I believe you were the one with most interest in this feature

[sjrl]

Thanks for the continued work on this @davidsbatista! I have a few more comments:

- It doesn't seem like calculating the page_number has made it yet, is that right? Update: Sorry I was looking at an older commit.

• Also my impression is that there is a lot of similar functionality between the DocumentSplitter and the RecursiveDocumentSplitter and I wonder if it would make sense to refactor one or both so they could use similar functionality. Otherwise it seems like we have a lot of similarish functions doing the same thing which also requires double the the tests. What do you think?

Hey @davidsbatista just pinging to make sure you saw this.

[davidsbatista]

Thanks for the continued work on this @davidsbatista! I have a few more comments:
- It doesn't seem like calculating the page_number has made it yet, is that right? Update: Sorry I was looking at an older commit.

• Also my impression is that there is a lot of similar functionality between the DocumentSplitter and the RecursiveDocumentSplitter and I wonder if it would make sense to refactor one or both so they could use similar functionality. Otherwise it seems like we have a lot of similarish functions doing the same thing which also requires double the the tests. What do you think?

Hey @davidsbatista just pinging to make sure you saw this.

I saw it as well, no worries - I need to still to make the edge cases work and add the new feature and then look into this - the changes and needed features started to pile up, I need more time - but I did not forgot no worries

[davidsbatista]

Also my impression is that there is a lot of similar functionality between the DocumentSplitter and the RecursiveDocumentSplitter and I wonder if it would make sense to refactor one or both so they could use similar functionality. Otherwise it seems like we have a lot of similarish functions doing the same thing which also requires double the the tests. What do you think?

So, I agree and we should definitely do it. But, out of the scope of this PR/issue - we should finalise this one first, merge it and then see how to extract common functionalities from both to an external utils file/module.

[sjrl]

Also my impression is that there is a lot of similar functionality between the DocumentSplitter and the RecursiveDocumentSplitter and I wonder if it would make sense to refactor one or both so they could use similar functionality. Otherwise it seems like we have a lot of similarish functions doing the same thing which also requires double the the tests. What do you think?

So, I agree and we should definitely do it. But, out of the scope of this PR/issue - we should finalise this one first, merge it and then see how to extract common functionalities from both to an external utils file/module.

Okay sounds good let's do that. Could you open an issue to track this?

[davidsbatista]

Okay sounds good let's do that. Could you open an issue to track this?

Issue open: #8645