Chunking

When importing a 100-page internal manual into a RAG system, you can't just pass it to the LLM as-is. Beyond context window limitations, search accuracy suffers significantly. Chunking is a preprocessing technique that splits documents into appropriately sized fragments (chunks) to achieve optimal granularity for both retrieval and generation.

Why Splitting Is Necessary

In RAG's retrieval step, documents semantically close to the user's question are found. But if an entire document is converted into a single vector, the characteristics of individual topics get diluted and search accuracy drops dramatically. For example, if you want to know "how to apply for paid leave" but the entire employment regulations are in one chunk, the vector becomes a vague mix of payroll, attendance management, and other information. Proper splitting enables accurate retrieval of only the parts truly relevant to the question.

Fixed-Length Chunking vs. Semantic Chunking

The simplest method is fixed-length chunking, which mechanically splits by character or token count. A range of 500-1,000 tokens is a common guideline. It's easy to implement and fast to process, but carries the risk of cutting mid-sentence or breaking semantic coherence.

Semantic chunking, by contrast, considers semantic coherence when splitting. It detects paragraph and heading boundaries, and topic transitions to divide text into natural units, improving search accuracy. LangChain's RecursiveCharacterTextSplitter is a practical approach that attempts hierarchical splitting in the order of headings, paragraphs, sentences, and characters.

The Importance of Overlap

Adding overlap (redundant portions) between chunks is also an important strategy. For example, giving 500-token chunks an overlap of 50-100 tokens preserves context near split boundaries. Because the end of the previous chunk and the beginning of the next chunk overlap, information loss across boundaries can be prevented.

The Chunk Size Trade-off

Chunk size selection involves trade-offs. Smaller chunks (200-300 tokens) improve search accuracy but tend to lack necessary context. Larger chunks (1,000-2,000 tokens) contain rich context but also include irrelevant information and increase token costs. Since the optimal size varies by data characteristics and use case, testing and adjusting with actual queries is the practical approach. Using smaller chunks for FAQ-style content and larger chunks for technical documentation is an effective strategy.