Production RAG - Part 6: Entities and Knowledge Graphs

At some point in every RAG project, a category of question stops working and no amount of tuning fixes it. Better chunking does not help. A stronger reranker does not help. The reason is that two completely different failures both present as “the system could not answer”, and only one of them is a retrieval-quality problem at all.

Part of the Production RAG series (9 parts)

1. Production RAG - Part 1: Anatomy of the Pipeline
2. Production RAG - Part 2: Extracting Trustworthy Text from PDFs
3. Production RAG - Part 3: Chunking for Retrieval
4. Production RAG - Part 4: Hybrid Search, Fusion and Reranking
5. Production RAG - Part 5: Images and Multimodal Retrieval
6. Production RAG - Part 6: Entities and Knowledge Graphs ← you are here
7. Production RAG - Part 7: Grounded Generation
8. Production RAG - Part 8: Evaluation and Operations
9. Production RAG - Part 9: Splitting an Answer into Claims

Disclaimer. This post is drafted with assistance from large language models (Claude Opus 4.8 and DeepSeek V4 Pro) based on conversations exploring production RAG concepts. All content has been reviewed, edited, and verified by a human author.

Two failures, identical symptoms, opposite fixes. Diagnosing which one you have is the decision point for everything below.

Matching failure. The answer sits inside one chunk, but the chunk shares no vocabulary with the query — “who can approve prod deploys?” against a chunk titled “release sign-off authority”. Retrieval never surfaces it. The fix is chunk enrichment.

Reachability failure. No single chunk contains the answer. It spans an incident report, a runbook and an ownership table, joined by a shared site_id. No retriever that scores chunks independently can assemble it, because no individual chunk is a good match for the whole question. The fix is graph traversal.

What entity extraction buys you

Named-entity recognition tags spans of text as typed entities — products, services, regions, error codes, versions. For retrieval it buys three concrete things:

• Precision filtering. “EKS pricing in eu-west-1” can filter candidates to chunks tagged service=EKS AND region=eu-west-1 before similarity scoring runs at all — the metadata filter from part 1, generated automatically.
• Disambiguation. The token Lambda means something different in a physics corpus and an AWS corpus. Typed entities carry what embeddings blur.
• Graph edges. Entities are the nodes. No entities, no graph.

The pipeline is only as good as its weakest stage: extraction feeds normalization feeds indexing feeds retrieval. A 90%-accurate extractor with no normalization underperforms an 80% extractor with proper alias collapse, because the errors compound multiplicatively down the chain.

Model tiers and what they cost

Extraction runs in two places with very asymmetric budgets. At ingest, per chunk, once — you can afford a heavy model. At query time, per request, latency-critical — you cannot. Both must land on the same canonical IDs, or your filters silently match nothing.

Tier	Examples	Character	Fits
Rules / gazetteer	Regex, Aho-Corasick over known names	Free, exact, zero recall beyond the list	Codes, IDs, SKUs
Classic NER	spaCy en_core_web_sm	Fast, generic types only	People, orgs, places
Zero-shot span models	GLiNER family	Arbitrary type labels at inference, no training	Sweet spot for custom domains
LLM extraction	Small hosted models	Best judgment; slow, priced per token	Relations, hard cases, teaching

Log scale. Self-hosted tiers sit one to three orders of magnitude below hosted LLM extraction.

The pattern that captures most of the value: use an LLM once as a teacher, not forever as a worker. Have a hosted model label 5–20k chunks, fine-tune a small span model on those labels, then run the student self-hosted across the corpus and at query time. You pay LLM prices for a bounded labelling job instead of an unbounded extraction bill.

A small span model exported to ONNX with int8 quantization runs comfortably on CPU — roughly 150–300MB resident, tens of milliseconds per chunk. One operational note that saves real debugging time: pin intra-op threads (OMP_NUM_THREADS=1 per worker) and scale by process, not by ONNX thread count. Oversubscription makes inference slower under concurrency, which looks exactly like a model problem and is not.

Normalization is the whole game

Raw extractions are surface forms. Retrieval needs identities.

The query-time extractor must land on the same canonical ID, whichever surface form the user happened to type.

CREATE TABLE entities(
  id TEXT PRIMARY KEY,        -- 'aws:eks'
  type TEXT, canonical TEXT,  -- 'service', 'Amazon EKS'
  aliases TEXT                -- JSON: ["EKS","Elastic Kubernetes Service","k8s cluster"]
);
CREATE TABLE chunk_entities(
  chunk_id INTEGER, entity_id TEXT, count INTEGER,
  PRIMARY KEY(chunk_id, entity_id)
);

Wire entities into retrieval in increasing order of aggression: boost (add score weight to candidates sharing the query’s entities — safe), filter (restrict candidates to matching chunks — precise, brittle), expand (add alias surface forms to the keyword query — a recall win). Start with boost. Graduate a query pattern to filter only once you have measured extraction precision for that pattern.

Filtering fails closed. One missed extraction at query time and the filter excludes every relevant chunk — the user gets an empty answer, and nothing errored. Always carry a fallback: if the filtered candidate set is empty or suspiciously small, rerun unfiltered.

Chunk enrichment

For matching failures, enrichment appends synthetic retrieval handles to a chunk at ingest time: a one-line summary, likely user questions, alias expansions, canonical entity names. The handles are indexed. They are never displayed.

Concatenating the handles into the chunk’s one stored string works, and then pollutes citation. Concatenating them into one indexed column fixes citation and moves the damage into ranking, where a machine-written question outbids the prose that actually answers the question. Split at the schema level instead, and give each part its own column:

ALTER TABLE chunks ADD COLUMN display_text   TEXT;  -- verbatim, user-facing
ALTER TABLE chunks ADD COLUMN heading_path   TEXT;  -- deterministic
ALTER TABLE chunks ADD COLUMN overlap_prefix TEXT;  -- deterministic
ALTER TABLE chunks ADD COLUMN enrichment     JSON;  -- {summary, questions, aliases}
ALTER TABLE chunks ADD COLUMN enrichment_fts TEXT;  -- the same, flattened for FTS5

UPDATE chunks SET display_text = text WHERE display_text IS NULL;

-- the FTS5 from part 4 indexed one column; rebuild it with one column per source
DROP TABLE IF EXISTS chunks_fts;
CREATE VIRTUAL TABLE chunks_fts USING fts5(
  body,              -- fed from display_text
  enrichment,        -- fed from enrichment_fts
  heading_path,
  overlap_prefix,
  tokenize = "porter unicode61 tokenchars '-_'"
);

Enrichment is stored twice on purpose: the FTS insert wants one flat string, while the embedding step below needs to pull the summary back out on its own.

Separate columns exist so that bm25()’s arguments — the per-column weights from part 4 — can hold synthetic text in its place:

-- still ORDER BY ascending: FTS5 scores are negated, as in part 4
SELECT rowid, bm25(chunks_fts, 1.0, 0.35, 0.6, 0.15) AS score
FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY score LIMIT 50;

Column	Weight	Reasoning
body	1.00	Real prose is the baseline
heading_path	0.60	Strong topical signal, very few tokens
enrichment	0.35	Enough to rescue, not enough to outbid
overlap_prefix	0.15	Duplicated from a neighbour; suppresses double-hits

Enrichment’s 0.35 is the knob that matters. A synthetic question contributes about a third of what real prose does — enough to surface a chunk that would otherwise never appear, not enough to beat a chunk that genuinely contains the user’s words.

Index text and display text must be different columns. Every synthetic addition — overlap, heading ancestry, enrichment — improves retrieval and pollutes citation. The moment you concatenate them into one stored string, users start seeing repeated sentences and machine-written questions inside quoted evidence. This is the load-bearing schema decision of the entire ingest side, and it is nearly free if you make it early and painful to retrofit if you do not.

The dense leg does not get the same input

All of the above is the lexical arm. The embedding is a string you assemble in code and hand to the model — it never exists as a column — and it should not be the same string. Feed it the heading path, the summary and the body. The questions and aliases stay out.

Aliases are exact-match fodder; a list of surface forms means nothing to an embedding model and only adds noise. Questions are worse. Three of them plus a paragraph of prose produce a vector that is part what a user might ask and part what the document says, and matches nothing especially well — you have made the chunk slightly wrong for everything.

If matching failures really are your dominant loss, the answer is to keep the questions apart rather than blend them in: embed the body once and each question once, all rows pointing at the same chunk id, then deduplicate by chunk id after the search. That is one extra vector per question on top of the body — three to four times the rows, and the same multiple on index size — in exchange for preserving each vector’s precision, which concatenating them destroys all at once.

Graph-hybrid retrieval

For reachability failures, store typed edges alongside the chunks and traverse at query time.

CREATE TABLE nodes(id TEXT PRIMARY KEY, type TEXT, label TEXT);
CREATE TABLE edges(
  src TEXT, rel TEXT, dst TEXT,
  chunk_id INTEGER,                       -- provenance: which chunk asserted this edge
  PRIMARY KEY(src, rel, dst, chunk_id)
);

The flow: extract entities from the query, seed nodes, expand one or two hops, collect the chunks that assert the traversed edges, and merge that chunk set into the fusion pool alongside the BM25 and vector candidates. Provenance on every edge is non-negotiable — the graph finds the chunks, but the chunks remain the evidence the answer cites.

The hub problem. One node — aws, prod, production — connected to everything makes two-hop expansion return half the corpus. Cap fan-out per node, penalize high-degree nodes during expansion, or drop hub-typed nodes from traversal entirely. Without this, graph retrieval degrades into noise precisely on the biggest corpora, which is where you wanted it most.

Before committing to typed edges, try the cheap middle ground: co-occurrence linking, connecting entities that appear in the same chunk with no relation typing at all. It answers “what is connected to X?” surprisingly well, costs one SQL join, and its failures tell you whether typed relations are worth extracting.

Automating graph construction

Hand-building a graph does not scale. Extracting triples with an LLM and trusting them does not survive contact with reality. The pipeline that works has five stages — and the emphasis is misplaced in most write-ups, because linking, not extraction, is where graphs die.

With entities fragmented across surface forms, reachable facts collapse geometrically with hop count. At four forms per entity, a three-hop question sees under 2% of the graph.

1 — Ontology first. A small typed schema — node types, relation types, domain and range constraints — written before extraction. Constrained output is dramatically cleaner than “extract entities and relationships”.

2 — Extraction with a span gate. Every triple must quote the text span that asserts it, and any triple whose span does not appear verbatim in the chunk is rejected. This single validation kills most hallucinated edges, and it costs one string comparison.

3 — Blocking. Entity resolution is pairwise-quadratic, so you never compare all pairs.

Blocking turns five billion comparisons into single-digit millions.

4 — Scoring and union-find. Score candidate pairs on name similarity, type agreement, alias overlap, embedding similarity and shared neighbours. Merge above a high bar using union-find so merges are transitive. Route the uncertain middle band to human review; below the band, leave them split.

5 — Incremental linking. New documents link into the existing graph using the same blocking and scoring against existing canonical entities, rather than triggering a rebuild.

The merge policy is asymmetric. A wrong merge — two real entities fused into one — silently corrupts every traversal through that node and is close to impossible to detect afterwards. A missed merge merely costs recall and is fixable the moment someone notices. Set the auto-merge threshold high, keep humans on the middle band, and prefer split over fused everywhere.

Build order for the whole entity and graph investment, cheapest first — and stop at the first level that fixes the failures you actually observe: gazetteer plus alias table → chunk enrichment → co-occurrence edges → typed relations from the span-gated extractor → full entity-resolution machinery.

GraphRAG and global questions

Everything above answers local questions: facts near identifiable entities. A different class of query defeats chunk retrieval entirely — global, sensemaking questions such as “what are the dominant failure themes across this corpus?” No chunk contains the answer, because the answer is a property of the corpus as a whole.

An LLM builds the graph, communities are detected, summaries are written per community, and answering is map-reduce over relevant summaries.

An LLM extracts entities and relations with no predefined schema and builds the graph. Community-detection algorithms find clusters of densely connected entities. An LLM writes summaries for each community at multiple hierarchy levels, from narrow subtopics up to broad themes. At query time the engine selects the relevant community summaries, generates a partial answer from each, and a final call synthesizes them.

GraphRAG’s cost lives at indexing time, and it is steep — a published worked example prices a twenty-document sample at roughly $6, extrapolating past $500 for eighteen hundred documents, before a single query is asked. Reserve it for corpora where global questions are genuinely asked. For entity-anchored questions, the graph-hybrid retrieval above answers at a fraction of the cost. The two are complements, not competitors.

What comes next

The next post assumes retrieval finally works, and asks the question that then becomes urgent: is the answer actually supported by what was retrieved?

References

1. Ofer Mendelevitch and Forrest Sheng Bao, Hands-On RAG for Production, O’Reilly Media, 2026
2. Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization, Microsoft Research 2024 — github.com/microsoft/graphrag
3. Zaratiana et al., GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer, NAACL 2024 — github.com/urchade/GLiNER
4. Traag, Waltman and van Eck, From Louvain to Leiden: Guaranteeing Well-Connected Communities, Scientific Reports 2019
5. Christen, Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, Springer 2012