fix: apply overlap only once in RecursiveDocumentSplitter

[Diwak4r]

Related Issues

• fixes RecursiveDocumentSplitter with overlap emits chunks not present in the source #12281

Proposed Changes:

RecursiveDocumentSplitter._chunk_text chunks recursively: a split that still exceeds split_length is re-chunked with the next separator. When split_overlap > 0, the overlap was applied at every recursion level, so chunks produced by an inner recursion got the overlap prepended a second time by the enclosing level. The result is chunks that are not substrings of the source text (e.g. "Overview\nOverview\nOverview\nThis module handles ing").

This PR moves the overlap application out of _chunk_text and applies it exactly once, on the fully chunked list, in _run_one (after the trailing-empty-chunk cleanup). Recursion boundaries are unchanged.

How did you test it?

Added a regression test (test_run_multiple_separators_with_overlap_applies_overlap_only_once) that chunks a multi-separator text with overlap and asserts every chunk is a substring of the source text. The test fails on the previous code (garbage chunks like "Overview\nOverview\nOverview\n..." are produced) and passes with this fix.

Ran the full test_recursive_splitter.py and test_type_utils.py suites: 1122 tests pass, including all existing overlap tests (test_apply_overlap_*, test_run_split_document_with_overlap_character_unit, test_run_split_document_with_overlap_and_fallback_character_unit, test_run_custom_sentence_tokenizer_document_and_overlap_char_unit, and the word/token overlap + _split_overlap metadata tests).

Notes for the reviewer

The bug only manifests when multiple separators cause actual recursion (single-separator configs are unaffected, which is why the existing tests did not catch it). _apply_overlap is a single-pass transform over a flat chunk list, so applying it once to the final list is equivalent to the previous behavior for the non-recursive cases.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@Diwak4r is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[HaystackBot]

Hi @Diwak4r, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[bogdankostic]

@Diwak4r Please sign the CLA for me to be able to review and merge this PR. Thanks!

[ebarkhordar]

Ran the invariant this fixes across randomized inputs in a clean python:3.11-slim container (pip install -e .), at 1ff39f52 and at 842519e5 on main. 120 random texts assembled from \n\n, \n and space separators, sweeping split_length and split_overlap, asserting that every emitted chunk is a substring of the source document:

split_unit    configs   main   this PR
char              720    263         0
word              720     44         0
token             480     97         0

test/components/preprocessors/test_recursive_splitter.py is 56 passed at 1ff39f52.

Two notes.

The three lines this removes from the fallback path in _chunk_text are the fix from #11768 (7ded47fc2). Its regression tests test_fallback_overlap_char_unit, _word_unit and _token_unit are inside those 56 and still pass from the new call site in _run_one, so the path is covered. It is the first thing to re-check if this placement is ever revisited.

The new regression test pins one hand-built example (exact chunk strings, len(chunks) == 6) on split_unit="char" only, so it will also fail on a future boundary change that has nothing to do with this bug. The property it is really pinning is the one in the table above, and it holds for all three units, so a loop over them asserting containment would catch the same class without freezing the boundaries. Happy to post the generator I used if that is useful.

For word units the overlap is rejoined with a single space in _create_chunk_starting_with_overlap, so the plain substring check does not hold there and I asserted instead that the chunk's word sequence is contiguous in the source. I did not check split_unit="sentence".