Chunking Guide: Experiments

Code, notebooks, and results behind the guide How to Choose a Chunking Strategy (guide-draft.md).

The guide's claim is that two questions decide most of a chunking setup:

1. Context budget: how many tokens of retrieved text you can send to the LLM per answer.
2. Context dependency: how much a chunk needs the rest of its document to make sense.

Every experiment here compares setups at the same token budget, so a setup that sends more text gets no credit for it. The main metric is evidence recall at budget ("found rate"): the share of questions whose labeled answer passage is in the text sent to the LLM. A passage counts as found when at least 80% of its characters are in the sent text (FOUND_AT in metrics.py).

What's in the repo

Path	What it is
notebooks/	One notebook per experiment, numbered 00 to 08. Start here.
methods.py	Chunking boundaries, retrieval modes, and run_setting, which runs one setting on one dataset. GRID lists every setting.
data.py	Loaders for the three datasets. Each returns documents, questions, and the character span of each answer passage.
metrics.py	Found rate at a budget and bootstrap confidence intervals.
settings.py	Embedding model (BAAI/bge-small-en-v1.5), token budgets (500, 1k, 2k, 4k, 16k), and which datasets search the whole collection or only their own document.
config.py	Reads .env, creates Qdrant and LLM clients.
agent.py	The grep agent for notebook 06.
dependency.py	The stand-alone test for notebook 02.
paraphrase.py	Rewrites questions in other words for notebook 07.
summaries.py, contextual.py	LLM-written document summaries and chunk contexts for notebook 08.
check_chunking.py	Standalone script readers run on their own documents (notebook 05).
results/	Saved output of every run, as JSON. Notebooks read from here, so reruns only compute what is missing.
results-summary.md	All result tables in one place.
guide-draft.md, chunking-guide-outline.md	The guide draft and its outline.
chunking-guides-survey.md	Survey of 47 published chunking guides that motivated the experiments.

Datasets

All three have human-labeled answer passages, so recall is measured exactly instead of judged by a model.

Dataset	Documents	Why it is here	Search scope
SQuAD v1.1 (validation)	Wikipedia articles, about 8,000 tokens each	Passages mostly stand on their own	Whole collection
RepliQA (split 0)	Policies, incident reports, and news about invented entities, about 1,250 tokens each	No LLM can answer from training data	Whole collection
Qasper (validation and test)	NLP research papers, about 5,000 tokens each	Passages depend on their paper. Questions were written without seeing the paper	Within the question's own paper

Qasper searches inside one paper because its questions assume the paper is already open. Searching all 697 papers rarely finds the right one (see results/early/README.md).

SQuAD and RepliQA annotators wrote questions while reading the paragraph, so questions reuse its words. That favors keyword search. Notebook 07 measures how much.

The baseline and the settings

The baseline is LangChain's recursive splitter, 256 tokens, no overlap, no prefix, dense search with bge-small-en-v1.5. Each setting in methods.GRID changes one thing from it. Result files are named results/grid_<dataset>_<setting>.json.

Setting	What changes
baseline	Recursive, 256 tokens, dense.
size_128, size_510	Chunk size. 510 is the largest that fits bge-small's 512-token input with its 2 special tokens.
fixed	Fixed windows of 256 tokens that ignore structure.
sections	Split at document headings first, then recursively inside each section.
semantic	Cut where neighboring sentences differ most in embedding (top 5% of distances), then split any group over 256 tokens.
overlap_64	64 tokens of overlap between neighboring chunks.
prefix	Prepend the document title and nearest section heading to each chunk before embedding.
neighbors	Send each retrieved chunk with the chunk before and the chunk after it.
parent	Search 128-token chunks, send the 510-token chunk that contains each one.
bm25	Keyword search (BM25 sparse vectors) instead of dense.
hybrid	Dense and BM25 combined with reciprocal rank fusion.
grep	No index and no LLM. Ranks lines by how many question words they contain (whole words, case-insensitive) and sends each with one line around it.
sections_128, hybrid_128, hybrid_sections	Combinations of the strongest single changes.
summary_prefix, llm_context	Prepend an LLM-written summary or chunk context (notebook 08).
prefix_raw, summary_raw, llm_context_raw	Same as above, but embed the chunk with its added context and send only the raw chunk text.

Suffixes on result files mark a different question set: _sample300 and _paraphrased300 (notebook 07: the same 300 questions in original and reworded form) and _papers100 (notebook 08: Qasper limited to 100 papers).

Notebooks

Run them in order the first time. Notebooks 01, 03, and 04 need no LLM calls.

Notebook	Guide section	Question it answers	Setup
00_setup	n/a	Do the datasets load, and do chunks fit the embedding model?	Downloads the datasets into data/. Checks that the longest chunk is within bge-small's 512-token limit.
01_baseline	0, 1	What does a default setup find at each budget? What would sending the whole document cost?	Runs baseline on all three datasets. Reports found rate at 500 to 16k tokens and average tokens per document.
02_dependency	2	How much do answer chunks depend on their document?	For 100 random questions per dataset, an LLM reads the answer chunk without its document and checks four failure types from the ConTEB benchmark. Done plain and with the title prefix. A person labels 30 chunks in results/dependency_spotcheck.csv, and a second LLM from another family checks agreement.
03_methods	0, 3	Which single change helps at which budget?	Runs every setting in GRID except the LLM-context ones on all three datasets. One table per dataset, with confidence intervals.
04_recommendations	4	Does the recommendation transfer to a new dataset?	Leave-one-dataset-out. Picks the best average-rank setting on two datasets and checks it lands within 5 points of the best on the third. Grep is excluded.
05_validation	5	Can a reader run this on their own documents?	Runs check_chunking.py on 40 RepliQA documents as a worked example.
06_grep_agent	0	Is a grep agent a substitute for chunking?	An LLM with grep and read tools searches for up to 10 steps. 100 questions per dataset, original and reworded. Reports found rate, tokens per answer, and steps. Then repeats RepliQA with a weaker model (Llama 3.1 8B) and a stronger one (Claude Haiku 5.5).
07_question_bias	0	Do results hold when questions do not reuse the passage's words?	An LLM rewrites 300 SQuAD and 300 RepliQA questions, keeping names and numbers. Runs baseline, size_128, bm25, hybrid, and grep on both versions. Qasper is skipped.
08_llm_context	3	Is LLM-written context worth its indexing cost?	Compares baseline, title prefix, summary prefix, and per-chunk context (as in Anthropic's Contextual Retrieval), each sent to the LLM and embedded only. Qasper uses 100 papers to limit cost. Reports indexing calls and tokens.

Run the experiments

Requires Python 3.12 or later and uv.

uv sync
cp .env.example .env     # then fill in OPENROUTER_API_KEY
uv run jupyter lab notebooks/

.env settings:

• OPENROUTER_API_KEY: needed for notebooks 02, 06, 07, and 08. LLM calls go through OpenRouter.
• LLM_MODEL: the model for those notebooks. The saved results use deepseek/deepseek-v4.1-flash, with anthropic/claude-haiku-5.5 as the second judge and as the stronger agent model.
• QDRANT_URL: a folder path (default ./qdrant_storage) runs Qdrant in local mode, with one subfolder per collection. An http(s) URL connects to a server, and then set QDRANT_API_KEY.
• QDRANT_COLLECTION_PREFIX: collections are named <prefix>_<dataset>_<chunking>_<vectors>.

Notes on cost and reruns:

• Embeddings are cached in cache/ and results in results/, so reruns skip finished work. Delete a result file to recompute it.
• Notebook 08 is the expensive one: about 26,500 chunk-context calls and 60M input tokens, about $11 with DeepSeek v4.1 Flash.
• notebooks/build.py regenerates notebooks from cell sources. It keeps existing notebooks and their outputs unless you name them: uv run python notebooks/build.py 03_methods.ipynb.

Check chunk sizes on your own documents

check_chunking.py is a standalone script (dependencies are declared inline, so uv runs it directly). You give it:

• a folder of .txt or .md files
• a CSV with columns question, doc (a file name in the folder), and passage (text copied from that file that answers the question)

./check_chunking.py docs/ questions.csv --sizes 128,256,510 --budgets 1000,4000

It prints recall at each chunk size and budget, tokens sent per answer, and a warning if any chunk is longer than the embedding model's input limit. It then sorts the misses at the best size by cause and says what to try:

• split: a chunk boundary cuts the answer. Split at headings, or use a larger size.
• wrong part: the right document was sent, but not the answer. Use smaller chunks.
• reworded: the question shares few words with the passage. Keep dense search.
• not found: the question shares words with the passage but the document was missed. Add BM25 (hybrid search).

Use --qdrant-url to index into a server instead of memory, and --model and --limit for a different embedding model.

Limits to keep in mind

• One embedding model (bge-small-en-v1.5). The input-length point applies to any model, but the ranking of settings might not.
• Small budgets are noisy on Qasper, and differences under about 5 points on the 300-question sets are within noise.
• Dependency labels come from LLMs. A person checked 30 chunks per judge and was stricter than both models, so the guide reports a range.
• overlap_64 results were regenerated after a fix to chunk span lookup for overlapping chunks (commit a77657b). Other settings use no overlap and were not rerun.