Why layout matters for RAG chunking
Retrieval quality is bounded by chunk quality. A chunker can only split along boundaries the extractor preserved.
- Plain-text OCR flattens the page: two columns interleave line by line, table cells become a word soup, and headers detach from their sections. Chunks then cut mid-thought, and retrieval returns fragments that embed poorly and read worse.
- Structured extraction keeps semantics: headings, paragraphs, lists, and tables arrive as labeled elements, so chunking can respect section boundaries and keep a table with its caption.
- Tables are the acid test: "Q3 revenue: 4.2M" only exists if row and column headers stay attached to the cell. Flattened tables are the single most common cause of confidently wrong RAG answers over financial and compliance PDFs.
- Reading order is retrieval order: sidebars, footnotes, and stamps injected mid-paragraph poison the embedding of otherwise clean chunks.
The honest tradeoff table
- Throughput: a lightweight OCR engine on GPU can be several times faster per page than a full layout pipeline running detection, table-structure, and reading-order models. Over a million-page backfill, that is days versus weeks of compute.
- Operational weight: general OCR is one model pair; document pipelines chain multiple models and need more memory and more tuning surface.
- Failure modes differ: general OCR fails by losing structure silently; document pipelines fail by mis-segmenting unusual layouts, which is at least visible in output structure.
- Handwriting humbles both: neither class is reliable on cursive; dedicated handwriting models or human review remain the honest answer.
- Cost of being wrong: re-running ingestion after discovering flattened tables means re-embedding the corpus. Structure errors are far more expensive to fix downstream than upfront latency.
A sane enterprise recipe
- Route, do not choose: classify pages first. Born-digital text needs no OCR; clean text-only scans go to the fast engine; anything with tables or multi-column layout goes to the structured pipeline.
- Measure on your documents: published benchmarks rarely include your scanner artifacts, stamps, and templates. Build a 200-page golden set with known answers and score end-to-end retrieval, not just character error rate.
- Chunk on structure: with structured output, split on headings and keep tables atomic with captions; with plain text, at minimum split on detected blank-line gaps rather than fixed character counts.
- Keep provenance: store page number and bounding box per chunk so answers can cite the exact region of the scan.
Metrics that actually predict RAG quality
- CER/WER: character and word error rate measure transcription, not structure. Necessary, wildly insufficient.
- TEDS: Tree Edit Distance Similarity scores table-structure recovery; the metric that correlates with answers over tabular PDFs.
- Reading-order accuracy: fraction of element pairs in correct sequence; predicts chunk coherence.
- End-to-end answer accuracy: the only metric leadership should see: same retriever, same LLM, swap only the extractor, and grade answers on the golden set.
Rule of thumb from practice: when documents contain tables, extractor choice moves end-to-end answer accuracy more than swapping the embedding model does.