RAG Chunking Strategies: Fixed vs Semantic vs Recursive

RAG Chunking Strategies: Fixed vs Semantic vs Recursive

Photo by AltumCode on Unsplash

Quick Answer: Start with recursive character splitting at 400-600 tokens with 10-15% overlap — it wins or ties in most production RAG evaluations while costing almost nothing. Upgrade to structure-aware (markdown/heading-based) chunking for docs, parent-document retrieval for support content, and contextual retrieval (prepending an LLM-generated context line to each chunk) when you need a 20-35% reduction in retrieval failures and can afford one cheap LLM call per chunk. Pure semantic chunking is rarely worth 10-50x the preprocessing cost. Measure everything with recall@k on a 50-200 question golden set before believing any of it.

On This Page

• Why Chunking Drives RAG Quality
• The Seven Strategies Compared
• Fixed-Size and Recursive Splitting
• Semantic and Structure-Aware Chunking
• Late Chunking and Parent-Document Retrieval
• Chunk Size and Overlap by Content Type
• Contextual Retrieval and Metadata Enrichment
• How to Evaluate Your Chunking
• Frequently Asked Questions

Why Chunking Drives RAG Quality

Chunking is the highest-leverage, lowest-glamour decision in your RAG pipeline. Here's why it dominates:

Embeddings compress. A 1024-dim vector can faithfully represent maybe a paragraph of meaning. Feed it a 3,000-token chunk and the embedding becomes a blurry average — your "how do I rotate API keys" query won't match the one relevant paragraph buried inside a chunk about account settings.

Chunks are what the LLM sees. Retrieval hands chunks, not documents, to the generator. Split a table from its header row, or an answer from the sentence that scopes it ("this applies only to enterprise plans"), and the LLM confidently hallucinates.

Errors compound downstream. Teams spend weeks tuning rerankers and prompts to compensate for bad chunks. In post-mortems of RAG failures, retrieval — not generation — is the culprit in roughly 60-70% of cases, and chunking is the most common root cause.

"Every RAG quality issue we triage starts the same way: show me the chunks. Nine times out of ten, the model never had a chance." — Applied AI Engineering newsletter, Q2 2026

The Seven Strategies Compared

The pattern across published evaluations and production systems in 2026 is consistent: recursive splitting is a brutally strong baseline, structure-aware methods win on structured content, and expensive semantic chunking only pays off on messy, topic-drifting corpora like meeting transcripts and crawled web pages.

Fixed-Size and Recursive Splitting

Fixed-size chunking cuts every N tokens regardless of content. It's what you get with three lines of code, and it's fine for benchmarking — but it happily splits sentences mid-clause, separates table headers from rows, and cuts code functions in half.

Recursive character splitting (the LangChain RecursiveCharacterTextSplitter pattern, reimplemented everywhere) fixes the worst of this cheaply. It tries to split on (paragraphs) first, falls back to , then sentences, then words, until chunks fit the target size. Structure is respected when it exists, and you always get bounded chunk sizes.

Practical defaults that survive contact with production:

1. Target 400-600 tokens per chunk for general content with modern embedding models.
2. Overlap 10-15% (50-80 tokens). Overlap is insurance against boundary-straddling answers; more than 20% mostly buys duplicate retrievals and higher storage.
3. Split on tokens, not characters — token counts are what your embedder and context window actually care about.
4. Never let a chunk cross a document boundary, and strip boilerplate (nav text, footers) before splitting; it poisons embeddings.

Semantic and Structure-Aware Chunking

Semantic chunking embeds every sentence, walks through the document, and starts a new chunk when cosine similarity between adjacent sentences drops below a threshold (commonly the 5th-25th percentile of the distribution). The result: chunks that align with topic shifts rather than arbitrary sizes.

The catch is cost and variance. You embed the corpus twice (once per sentence for splitting, once per chunk for indexing), preprocessing takes 10-50x longer, and chunk sizes swing wildly — you still need max-size enforcement. In head-to-head evaluations it beats recursive splitting on some corpora by 3-8 points of recall and loses on others. Test on your data; don't assume.

Structure-aware chunking is the better upgrade for most teams. If your content has structure — markdown headings, HTML sections, docstrings, legal numbered clauses — split on it:

• Chunk at heading boundaries; keep each section intact up to a max size, then recursively split inside it.
• Prepend the heading path to every chunk: "Billing > Refunds > Annual plans: ...". This one-line trick disambiguates chunks that are meaningless out of context and reliably improves retrieval.
• Keep tables and code fences atomic. A split table is garbage; a split function is worse.
• For code, chunk at function/class boundaries using tree-sitter or AST parsing, never raw lines.

Photo by DeepMind on Unsplash

Late Chunking and Parent-Document Retrieval

Two newer patterns matter in 2026:

Late chunking flips the order of operations. Instead of split-then-embed, you run the entire document (up to 8K-32K tokens) through a long-context embedding model once, get token-level embeddings that have "seen" the whole document, then mean-pool contiguous spans into chunk vectors. Every chunk embedding is contextualized — the pronoun "it" in paragraph 12 carries the meaning established in paragraph 2. Jina v3 and other long-context embedders support this natively. It's the right tool for documents dense with cross-references, at the cost of a more complex pipeline.

Parent-document retrieval (small-to-big) decouples what you search from what you send to the LLM. You index small, precise chunks (150-300 tokens) whose embeddings are sharp, but when one matches, you return its parent — the surrounding section of 1,000-2,000 tokens. Precision at retrieval time, context at generation time. Most vector stores make this a metadata lookup (store parent_id on each small chunk). For support knowledge bases and legal documents, this pattern alone often outperforms every clever splitting algorithm.

Chunk Size and Overlap by Content Type

Numbers to start from, tuned in production RAG systems:

Two overriding rules: smaller chunks for precision-critical retrieval (citations, compliance), bigger chunks or parent-document for synthesis-heavy generation (summaries, comparisons). And your embedding model's sweet spot matters — most models degrade past ~1,000 tokens even when their context window is technically larger. Our embedding model comparison guide covers which models hold up at long chunk lengths.

Contextual Retrieval and Metadata Enrichment

The single most impactful chunking upgrade of the past two years is contextual retrieval, popularized by Anthropic's engineering write-up: before embedding each chunk, use a cheap LLM to generate a 50-100 token line situating the chunk within its source document, and prepend it.

A raw chunk like "Revenue grew 3% over the previous quarter" becomes:

This chunk is from ACME Corp's Q2 2026 10-Q filing, discussing
quarter-over-quarter revenue in the cloud services segment.
Revenue grew 3% over the previous quarter...

Anthropic's published numbers: contextual embeddings cut retrieval failure rates by 35%; combined with contextual BM25 and reranking, by up to 67%. The cost is one small-model call per chunk — with prompt caching (the full document sits in the cached prefix), roughly $1 per million document tokens. For most corpora under a few GB, that's lunch money for the biggest quality jump available.

Cheaper metadata enrichment that stacks with it:

• Attach filterable metadata to every chunk: source, date, product version, access level, document type. Pre-filtering (version = "v4") before vector search eliminates entire classes of wrong-document retrievals.
• Hybrid search: index chunks in BM25 alongside vectors and fuse results (reciprocal rank fusion). Keyword search catches exact identifiers — error codes, function names, SKUs — that embeddings fuzz over.
• Generated questions: for FAQ-style corpora, generate 2-3 hypothetical questions per chunk and embed those too, matching query phrasing to question phrasing.

How to Evaluate Your Chunking

Never ship a chunking change on vibes. The evaluation loop:

1. Build a golden set: 50-200 real user questions, each labeled with the document/passage that answers it. Pull from actual query logs, not invented questions.
2. Measure retrieval directly — before any LLM generation:

3. A/B chunking strategies on identical embeddings and retrieval settings — change one variable at a time.
4. Then check end-to-end answer quality with an LLM judge (RAGAS-style faithfulness and answer relevance) to catch cases where retrieval is fine but chunks lack context to generate from.
5. Re-run the suite on every corpus refresh; chunking that worked on last year's docs quietly rots as content changes. Automating this in CI takes an afternoon and pays compounding dividends.

A practical recipe to ship this week: recursive splitting at 500 tokens/12% overlap → add heading-path prefixes → add BM25 hybrid → measure. If recall@5 is below target, add contextual retrieval, then a reranker. Only then consider exotic splitting.

Related Reads

• Large-Scale Embedding Serving: Architecture, Indexing, and Retrieval

Key Takeaways

• Start with recursive character splitting at 400-600 tokens and 10-15% overlap—it’s the strongest baseline for most RAG pipelines, balancing cost and performance with minimal complexity.
• Use structure-aware chunking (e.g., markdown/heading-based) for documentation, wikis, or code—prepend heading paths to chunks and keep tables/code blocks atomic to improve retrieval accuracy by 20-35%.
• For support content or legal documents, adopt parent-document retrieval: index small, precise chunks (150-300 tokens) but return their larger parent sections (1,000-2,000 tokens) to combine retrieval precision with generation context.
• Implement contextual retrieval by prepending a 50-100 token LLM-generated summary line to each chunk before embedding—this reduces retrieval failures by ~35% at a cost of ~$1 per million document tokens.
• Evaluate chunking strategies rigorously with recall@5/20 on a 50-200 question golden set derived from real user queries; automate this in CI to catch performance drift as content evolves.
• Avoid semantic chunking unless your corpus is messy (e.g., transcripts, web pages)—its 10-50x higher preprocessing cost rarely justifies marginal gains over recursive or structure-aware methods.