TextDocumentCleaner

[ZanSara]

Cleans text to make it more readable, both by humans and by LLMs. It's task is to perform find/replace of some strings, both user-given or predefined.

Ideally it would be able to replace/remove:

• Whitespace, empty lines and other control characters
• Headers and footers (auto-detected repeated strings at the start and at the end of every page)
• User-specified strings
• User-specified regexes

TextDocumentCleaner should also update the structure dictionary properly if it’s present, so it should retain only the page numbers and headings that are still present after the cleaning and updating the chars positions.

Draft I/O

@component
class TextDocumentCleaner:

    @component.output_types(documents=List[Document])
    def run(self, documents: List[Document], ... clean_empty_lines, clean_whitespace, clean_substrings, etc ...):
        # cleans the documents
        return {"documents": documents}