Inconsistent behavior of CharacterTextSplitter when changing keep_separator for special regex characters

[mmz-001]

System Info

LangChain v0.0.225, Windows, Python 3.10

Who can help?

@hwchase17

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

The behavior for CharacterTextSplitter when changing keep_separator when using normal characters is like this:

text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator="_",
    keep_separator=False,
)

text_splitter.split_text("foo_bar_baz_123")
# ['foo', 'bar', 'baz', '123']

text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator="_",
    keep_separator=True,
)

text_splitter.split_text("foo_bar_baz_123")
# ['foo', '_bar', '_baz', '_123']

However, when using special regex characters like . or * the splitter breaks when keep_separator is False.

text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator=r"\.",
    keep_separator=False,
)

text_splitter.split_text("foo.bar.baz.123")
# ['foo.bar.baz.123']

The special characters should be escaped, otherwise it raises an error. For example, the following code raises an error.

text_splitter = CharacterTextSplitter(
    chunk_size=4,
    chunk_overlap=0,
    separator=r"?",
    keep_separator=True,
)

text_splitter.split_text("foo?bar?baz?123")

I'll make a PR to fix this.

Also, the documentation never mentions that the separator should be a regex, I only found out the hard way after getting regex errors on one of the RecursiveTextSplitter splitters after updating LangChain. I think we should add a note about this in the documentation or the code.

Expected behavior

text_splitter = CharacterTextSplitter(
   chunk_size=4,
   chunk_overlap=0,
   separator=r"\.",
   keep_separator=False,
)

text_splitter.split_text("foo.bar.baz.123")
# ['foo', 'bar', 'baz', '123']

[dosubot]

Answer generated by a 🤖

Answer

Thank you for bringing this issue to our attention and for proposing a solution! It's great to see your willingness to contribute to LangChain.

Your proposed fix to handle special regex characters in the CharacterTextSplitter is definitely needed. It will ensure consistent behavior when changing the keep_separator parameter.

Additionally, your suggestion to add a note in the documentation or code about the separator being a regex is valuable. This will help users avoid confusion and potential errors.

Please go ahead and create a pull request with your proposed changes. The maintainers will review it and decide if it should be merged into the codebase.

Thank you for your contribution to LangChain!

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.