Document Chunking / Parsing

Before a document can be retrieved by a RAG system, it must first be readable. Enterprise documents are rarely clean: PDFs contain scanned pages, multi-column layouts, embedded tables, and headers that fragment badly into raw text. Legal contracts nest clauses within sections within annexes. Technical manuals have figures with captions, code blocks, and cross-references. Parsing transforms these heterogeneous formats into clean, structured text that preserves the document's logical organization.

Chunking is the subsequent step of dividing parsed text into segments sized appropriately for embedding and retrieval. The chunking strategy profoundly affects retrieval quality. **Fixed-size chunking** (e.g., 512 tokens with 50-token overlap) is simple but routinely splits sentences and concepts across chunk boundaries. **Semantic chunking** uses embedding similarity to find natural breakpoints where topic changes occur. **Hierarchical chunking** creates a tree structure — document → section → paragraph — enabling retrieval at multiple granularities and supporting techniques like HyDE (hypothetical document embeddings) and parent-child retrieval. **Agentic chunking** uses an LLM to determine semantically coherent boundaries, producing the highest quality but at significant compute cost.

The enterprise impact of chunking strategy is often underestimated. A well-known finding from production RAG deployments is that upgrading the chunking strategy from fixed-size to semantic or hierarchical can improve end-to-end answer quality by 15–40% without any changes to the embedding model or LLM. For document-heavy applications — legal contract analysis, policy search, technical documentation assistants — investing in a robust parsing and chunking pipeline typically delivers better ROI than spending the same effort on model upgrades.