Production RAG - Part 3: Chunking for Retrieval

A chunk is the smallest thing retrieval can return. You never get half a chunk, and never a chunk plus a bit of its neighbour. Whatever boundaries you draw at ingestion time are permanent — they define the set of answers your system is capable of producing at all. Which means a badly placed cut does not degrade an answer. It makes the answer impossible.

Part of the Production RAG series (9 parts)

1. Production RAG - Part 1: Anatomy of the Pipeline
2. Production RAG - Part 2: Extracting Trustworthy Text from PDFs
3. Production RAG - Part 3: Chunking for Retrieval ← you are here
4. Production RAG - Part 4: Hybrid Search, Fusion and Reranking
5. Production RAG - Part 5: Images and Multimodal Retrieval
6. Production RAG - Part 6: Entities and Knowledge Graphs
7. Production RAG - Part 7: Grounded Generation
8. Production RAG - Part 8: Evaluation and Operations
9. Production RAG - Part 9: Splitting an Answer into Claims

Disclaimer. This post is drafted with assistance from large language models (Claude Opus 4.8 and DeepSeek V4 Pro) based on conversations exploring production RAG concepts. All content has been reviewed, edited, and verified by a human author.

Neither chunk can answer the question. The information was in the corpus; the boundary removed it.

The goal of a chunker is therefore not “pieces of roughly 1,800 characters”. It is that every piece can stand alone.

Two judges, two different views

Chunks are selected by two mechanisms that look at completely different properties, and understanding both is what makes the rules below feel inevitable rather than arbitrary.

Vector search collapses a chunk to a single point in embedding space. A chunk containing two unrelated things lands somewhere between them, close to neither. A chunk stripped of context lands somewhere meaningless.

BM25 counts matching terms and divides by length. The length correction is necessary — otherwise long documents would always win — but it overshoots at the extremes. With FTS5’s defaults, the dl / avgdl term swings the denominator roughly fivefold between a 42-character fragment and a 1,780-character table.

Relative BM25 score for one identical term match at four chunk lengths. The orphan fragment nearly doubles the score of the full table it was cut from.

The intuition worth carrying: BM25 is asking “what fraction of this chunk is about your query?” A 42-character chunk entirely about 32x beats a 1,780-character chunk that is two percent about 32x. This single mechanism is behind two of the three chunking rules below.

The arithmetic behind the chart

With tf = 1 everywhere, IDF constant, and k1 = 1.2, b = 0.75, avgdl = 1200 (part 4 derives the formula these come from):

chunk	length	|D|/avgdl	relative score
orphaned table row	42	0.035	1.65
figure caption	400	0.333	1.38
prose paragraph	1,200 (average)	1.000	1.00
full table	1,780	1.483	0.83

The entire 2x spread comes from length normalisation alone. Make it concrete: a report produces four chunks, each containing Mumbai once. The orphan is | Mumbai | FY24 | 18,412 | 4.2% | — one severed row with no header, no units, no caption. The full table has twelve data rows and a header telling you 18,412 is in crores. Ask “what was Mumbai’s FY24 revenue?” and BM25 puts the only chunk that can answer it last.

The hidden assumption

Length normalisation makes BM25 ask what share of this chunk is the query term? That is a good proxy for aboutness when a human decided how long the text would be. Robertson and Zaragoza identified two reasons a document is long — verbosity (same content, more words) and scope (genuinely covers more ground) — and both point the same direction. The standard b = 0.75 was tuned on TREC collections: news wire, web pages, abstracts. Real documents with lengths their authors chose.

In a chunked corpus, length is a property of your splitter, not the author. Short chunks are usually the least useful — leftovers, split artifacts, rows orphaned at a boundary — and BM25 gives them the biggest bonus.

Saturation cannot rescue a long chunk

Can more matches make up the deficit? Hold length at 1,780 and vary the count:

matches	relative score
1	0.83
2	1.21
3	1.42
5	1.66

The full table needs five occurrences to edge past a single occurrence in the 42-character fragment. The ceiling is 2.2 and the orphan already sits at 1.65 — 75% of the theoretical maximum, from one match.

The compounding effect on avgdl

avgdl is computed over your index. Every orphan pulls it down, which makes ordinary chunks look proportionally longer, which penalises them harder. A chunker producing many fragments poisons the normaliser for everything else.

The strategy landscape

Five families, in rough order of sophistication:

Strategy	Idea	Weakness
Fixed-size	Cut every N characters, with overlap	Breaks sentences and structure; overlap adds redundancy
Sentence / paragraph	Cut on linguistic boundaries	Uneven sizes; still misses higher-level context
Recursive	Hierarchy of delimiters: paragraphs → sentences → clauses	Depends on delimiter quality; still ignores semantics
Document-structure	Cut on headings and sections	Needs well-structured documents
Semantic	Cluster sentences by embedding similarity	Expensive; unpredictable sizes; hard to debug

Two findings are worth knowing before you spend a sprint on the fancy end of that table.

First, published benchmark evidence (BEIR, RAGBench) shows fixed-size and semantic chunking performing indistinguishably on retrieval tasks. The caveat matters — those benchmarks contain short, pre-RAG-era passages, and the gap may well reopen on long documents — but it should make you spend the complexity budget where it demonstrably pays.

Second, when judging chunking through generation quality rather than retrieval metrics, remember the “lost in the middle” effect: LLMs attend most to the beginning and the end of their context. A chunking strategy can look worse purely because its chunks happened to rank into the middle of the context window. Do not conclude a strategy is better from generation metrics alone.

The chunker described below is document-structure chunking made table-aware. It captures the structural wins without semantic chunking’s cost.

Three content types, three different failures

Prose, tables and OCR text each violate “every chunk must stand alone” for a different reason. There is no single fix because there is no single failure.

Content type	How it breaks	Treatment
Prose	Meaning flows across the cut	~150-character overlap, snapped to word boundaries
Tables	Rows are meaningless without their header	Row-atomic; repeat header on every fragment
OCR text	Content is guessed, not read	Quarantined from overlap and heading inheritance

Prose: make the cut unclean

• Snap to a word boundary. Slicing at exactly 150 bytes cuts mid-word and mid-rune. A truncated token becomes a real FTS5 token that matches nothing. Walk back to the nearest space, and decode the last rune properly so multi-byte characters never split.
• Never overlap into a table. A tail landing inside a table prepends half a pipe row, which is broken markdown. Skip the overlap for that pair — a table chunk is already self-contained.
• Store the overlap separately from the body. Index heading_path, overlap_prefix and body — each in its own column, so part 6 can weight them differently — but display and cite only body. Retrieval gets the context benefit; the user never sees the same sentence twice.

Tables: two rules prose does not need

A table’s meaning is not in its rows — it is in the relationship between rows and header. So never cut mid-row (a half-row is corrupt, not merely incomplete), and repeat the header row on every fragment. A headerless fragment shows the answering model four bare numbers with no idea which one is a vCPU count and which is a price.

The subtler failure is the one the BM25 chart predicts. A table that spills 42 characters past the limit sheds a one-row runt, and that runt then outranks the table it came from on the very query it half-answers.

const (
    runtFrac     = 0.25 // a chunk under 25% of max is a runt
    overflowFrac = 1.25 // merging may push a chunk to 125% of max
)

func coalesceRunts(chunks []chunk, max int) []chunk {
    for i := len(chunks) - 1; i > 0; i-- { // backward: a merge can create a new runt
        if float64(chunks[i].chars) >= runtFrac*float64(max) { continue }
        if float64(chunks[i-1].chars+chunks[i].chars) > overflowFrac*float64(max) {
            continue // pathologically long - leave it alone
        }
        chunks[i-1] = mergeChunks(chunks[i-1], chunks[i])
        chunks = append(chunks[:i], chunks[i+1:]...)
    }
    return chunks
}

Accept a slightly oversized chunk rather than a runt. Then generalise the rule: enforce the 25%-of-max floor everywhere, with the single exception of a document that produces exactly one chunk. Any chunk below that floor is a length-normalization hazard regardless of what produced it.

Note the loop runs backward over the whole chunk list, not forward and not per-section. A mid-section table can shed a runt too, and merging one runt can make the previous chunk newly eligible.

Why backward matters: given chunks [1800, 1800, 300, 200], a forward sweep merges 300 into the preceding 1,800 to give [1800, 2100, 200], and then stalls. Absorbing the trailing 200 would take that chunk to 2,300, past the 2,250 overflow ceiling, so the guard refuses — and the runt survives the pass that exists to remove it. A backward sweep merges 200 into 300 first, giving [1800, 1800, 500], and 500 clears the floor. Backward lets the small pieces coalesce with each other before either one reaches for a full-size neighbour.

The forward sweep spends its merge budget on a chunk that did not need one. Sweep direction is not a style choice.

Re-check happens naturally: [1800, 100, 100] backward — the last 100 merges into the previous 100 giving 200, still under the floor, so it merges again into 1,800. Backward iteration handles this for free since you merge into position i-1 and examine it next.

The single exception is a document that produces exactly one chunk — an 80-character document has nowhere to merge to. Emit it, or you silently drop content.

The floor means maxChunkChars stops being a hard limit. Worst case, a full chunk absorbs a runt one character under the floor: 1,800 + 449 = 2,249. That is the real ceiling — both the embedder’s token limit and the per-chunk context budget need to tolerate it.

OCR text: quarantine, not repair

Prose and table chunks contain text read from the PDF. An OCR chunk contains text a vision model guessed from a picture. The risk is not incompleteness but wrongness — and overlap and heading inheritance are precisely the channels that would spread that wrongness into content you trust.

How it would leak	Rule that blocks it
OCR text bleeds into a clean neighbour	No overlap into or out of transcribed sections
An invented heading becomes an ancestor	Transcribed headings are never carried forward
Page number lost from the citation	Headings demoted so ## Page N survives

Channel 1 — overlap. Overlap copies a tail of one chunk onto the head of the next so a severed sentence is whole somewhere. When the source is guessed text, the copy lands in a trusted chunk — BM25 now scores that chunk on guessed tokens, and a citation points at a page that never said the thing. The rule blocks both directions: copying clean prose into the OCR chunk would make an image_transcript row partly untrue about its own type.

Channel 2 — heading inheritance. Overlap poisons one neighbour; a heading poisons everything beneath it. If the chunker prefixes each chunk with its section path, a heading the vision model invented is copied onto every chunk in the subtree — one bad guess, repeated N times, in the field that shapes both BM25 tokens and the embedding.

Channel 3 — page-anchor displacement. ## Page N carries citation provenance. If a transcribed heading is emitted at the same Markdown level, it closes the page section and becomes the nearest ancestor, displacing the page anchor. The chunk stays accurate and simply loses its page number. The fix is one line on the extractor side — force transcribed headings to at least one level below the page anchor:

level = max(PAGE_HEADING_LEVEL + 1, guessed_level)

The through-line across all three rules: reconstructed content is welcome in the index, but it is never allowed to contaminate the retrieval signals of content you trust. A single marker string — "(transcribed" — carries this from the extractor, through normalization, into the chunker, into the database row, and out to the answering prompt, where it lets the model hedge instead of stating a possibly-misread number as fact. One string, four consumers, no shared schema.

Block segmentation

The single most important implementation change from a naive chunker: chunk boundaries are only ever placed at block boundaries. Split the section body on blank lines, classify each block, and treat the block as the atom. A fourteen-row pricing table is one ~900-character block — packed whole, or split at row boundaries, but never at character 1,800.

Block type is recoverable from the assembled markdown itself, which keeps chunking a pure function of a string:

// A GFM table row is a pipe line whose NEXT line is the |---|---| separator.
// The two-line lookahead is what distinguishes a real table from prose with a pipe.
func isTableStart(line, next string) bool {
    return pipeLine.MatchString(line) && separatorLine.MatchString(next)
}

// A linearized table row: two or more 'Header: value' pairs joined by pipes.
var linearRow = regexp.MustCompile(`^[^|:]{1,40}: .+( \| [^|:]{1,40}: .+)+$`)

// OCR content announces itself in the heading the extractor wrote.
func isTranscribed(heading string) bool {
    return strings.Contains(heading, "(transcribed")
}

Resist the temptation to have the extractor emit typed spans instead. The moment extraction and chunking share a schema, every extractor change becomes a chunker change — and the OCR path has to fake structure it does not actually have.

Heading ancestry

Prefixing each chunk with its heading ancestry is one of the cheapest large wins available. Chunk three of a section, stripped of its heading, is a floating paragraph the embedding cannot place. Prepending Compute pricing > Page 12 > Reserved instances fixes that for both retrieval arms at once.

Maintain a heading stack with one extra bit per frame — inheritable bool, set false for transcribed headings. An OCR-synthesized heading labels its own chunks perfectly well, but a vision model’s invented heading for page 12 has no authority over native text on page 13.

Configuration and monitoring

knob	value	why
maxChunkChars	1,800	packing target, not a hard limit
chunkOverlapChars	150	enough to rejoin a severed sentence
runt floor	25% of max (450)	caps score inflation at 1.34x instead of 1.69x
effective max chunk size	2,249	max + floor - 1; assert on it
avgdl	~1,200	emergent, not configured — watch it drift

Log the chunk-length distribution after every ingestion run. Mean chunk length is nearly useless — if 3% of chunks are runts the mean barely moves. Track the p1 and p5 of the distribution, and separately the count of chunks below the floor. That count is an invariant: it should be exactly zero, except for single-chunk documents. If it goes positive, either the merge loop has a bug or a new document shape is defeating it.

What comes next

The next post moves to the query side: what these chunks actually get compared against, and why you need two retrievers plus a reranker rather than one very good one.

References

1. Ofer Mendelevitch and Forrest Sheng Bao, Hands-On RAG for Production, O’Reilly Media, 2026
2. Renyi Qu, Ruixuan Tu and Forrest Sheng Bao, Is Semantic Chunking Worth the Computational Cost?, EMNLP 2024
3. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024
4. Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021
5. SQLite FTS5 — the BM25 implementation
6. Robertson, S. and Zaragoza, H. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval, 2009