CAPTIVOLT INSIGHTS
Executive summary
Most RAG quality problems trace back to how documents were split. Chunk too small and the model loses the context to synthesise an answer; chunk too large and the real answer gets lost in the middle while the vector match blurs. Getting chunking right — overlap, metadata, and a size matched to your embedding model and content — is one of the highest-leverage things you can do for retrieval quality.
The problem
Splitting a document into naive fixed-size chunks breaks meaning at the boundaries: a clause is severed from the condition it depends on, a table is split across chunks and becomes unreadable, an answer is clipped halfway. The embedding model then indexes fragments that no longer mean what the author intended, and retrieval quietly degrades in ways demos never reveal.
A practical framework
- 01
Use a sliding window. Configure a chunk overlap of roughly 10–20% (for example 50–100 tokens on a 512-token chunk) so context carries across boundaries.
- 02
Enrich with metadata. Store the parent document, section title, and adjacent chunk IDs on each vector, so the system can pull surrounding context when an answer is clipped.
- 03
Match chunk size to the embedding model. The chunk must fit the model’s input limit — a 256-token model silently ignores anything past 256, however large you set the chunk.
- 04
Balance noise against context. Around 256–512 tokens is the general benchmark: too small and the model hallucinates for lack of context, too large and the answer is lost in the middle while the embedding turns generic.
- 05
Adapt to the content. Tables → smaller chunks wrapped in structural markers; legal and financial → larger chunks (512–768) so conditional clauses stay intact; support knowledge bases → small chunks (128–256) for specific questions.
- 06
When in doubt, use parent-child. Search on small child chunks for pinpoint accuracy, then return the larger parent chunk to the model for context — accurate retrieval and rich context together.
Key takeaways
- Most RAG failures are chunking failures.
- Overlap and metadata protect meaning at the boundaries.
- Chunk size must fit the embedding model’s token limit.
- Content type dictates strategy — tables, contracts, and FAQs each want different sizes.
- Parent-child chunking gives precise search with rich context.