RAG chunking strategies and how to pick one
The main RAG chunking strategies are fixed-size (cut every N tokens, usually with overlap), recursive or structural (split at headings, paragraphs and sentences), semantic (split where the meaning shifts, using embeddings), and parent-document (retrieve small chunks, hand the model their larger parent). For most business content, structural splitting is the best default: headings and paragraphs already mark the self-contained facts an AI answer can quote. Whatever strategy you pick, check the result - each chunk should still make sense when quoted alone.
The short answer
Retrieval systems split a page into chunks before they match and fetch them (what chunking is). The strategy is where the cuts go. The four families in real use:
| Strategy | How it cuts | Best for |
|---|---|---|
| Fixed-size | Every N tokens or characters, often with overlap | Uniform text with no clear structure |
| Recursive / structural | At the strongest separator available: headings, then paragraphs, then sentences | Documents that already have headings and sections |
| Semantic | Where the meaning shifts, judged by an embedding model | Dense prose, articles, mixed-topic pages |
| Parent-document | Small chunks for retrieval; their larger parent block is handed to the model | Long documents where precision and context both matter |
These are the families the main RAG guides describe: Pinecone, Databricks, LlamaIndex and Microsoft Learn all cover the same trade-offs: bigger chunks keep context but dilute relevance and cost; smaller chunks match queries precisely but lose the surrounding meaning.
Why structural splitting is usually the right default
Most business content is already chunked — by its author. Headings mark topics, paragraphs mark single ideas, and a well-written paragraph states its fact with its subject. Splitting at those boundaries respects the structure a human already built.
Fixed-size cuts ignore that structure and routinely cut a fact in half. Semantic chunking can recover meaning, but it depends on the embedding model's judgment, and the result still has to pass the same quotability test.
How to choose, in three questions
- Does the content have headings and paragraphs that each make a point? Use structural splitting.
- Is it uniform text without structure (transcripts, logs, generated lists)? Use fixed-size with overlap.
- Are the answers spread across long sections the model needs whole? Consider parent-document retrieval, which retrieves small chunks but hands the model the larger parent block (LangChain parent document retriever).
Chunk size follows from the same test: the smallest chunk that still contains one complete fact with its subject. Guides commonly start at a few hundred tokens with 10-20% overlap (Pinecone, Databricks).
The test every strategy has to pass
The strategy is only as good as the chunks it produces, because an AI answer can only quote what a retrieved chunk contains. Three failures show up regardless of technique:
- Dangling references — "as mentioned above, it costs $89", where "above" was never retrieved.
- Subjectless openings — a chunk that starts "Because of that, we recommend…" without naming the thing.
- Split facts — the price in one chunk, the service name in another.
NRLC.ai's studies of 395 AI answers about local businesses found the passages that actually got quoted were self-contained: a published price, a rating, a service in a named place (what AI assistants quote). No strategy paper promises that — it is a property of the text.
The checklist
- Pick the strategy by the document's structure, not by what a vendor guide used for its own demo corpus.
- Start at a few hundred tokens with 10-20% overlap, then adjust to the smallest self-contained chunk.
- Run the quotability test: read every chunk alone and fix the ones that do not name their subject and state their fact.
- Re-check after any template or content change — chunking is a property of the final HTML, not of good intentions.
Try it now: paste your page into the free chunking checker and read the chunks the way an AI would — alone.
Frequently asked questions
- What is the best chunking strategy for RAG?
- There is no universal best. Structural splitting (headings, paragraphs, sentences) is the best default for documents that already have structure. Fixed-size works for uniform text, semantic chunking for meaning-dense prose, and parent-document retrieval when you want small, precise chunks but full context in the answer.
- What chunk size should I use?
- Common starting points are 200-500 tokens or about 150-400 words, with 10-20% overlap between chunks. The right size is the smallest chunk that still contains one complete fact with its subject. If a chunk cannot be understood alone, it is too small or split in the wrong place.
- Does chunk overlap help?
- Overlap keeps context that a hard cut would split, such as a sentence continued across the boundary. It helps retrieval but does not fix a chunk that starts mid-thought or refers to "as mentioned above".
- Does semantic chunking improve AI citations?
- Semantic chunking can group related meaning better than fixed cuts, but it still produces chunks the AI must quote. A meaning-preserving chunk with a subjectless opening gets skipped just like any other. Measure the output, not the technique.
- How do I check my chunking strategy?
- Paste your content into the free Prechunk tool, read each chunk the way an AI would - alone - and fix the chunks that do not name their subject or state their fact in one place.