Document Chunking for RAG: Sizes, Overlap and Strategies

Sketch of document pages and overlapping chunks

Chunking is how you split documents into pieces that a retrieval-augmented generation (RAG) system can index, embed and retrieve. Start with sentence-preserving chunks of roughly 150 to 500 tokens for question answering, add a modest overlap, and test the result against your own questions. Then change the strategy only when the measurements tell you to. Fixed-size splitting is a fine first draft. For tables, manuals and multi-page reports, it usually isn’t the final answer.

This guide covers how to pick a chunk size, how much overlap to use, which strategy fits which document, and how to run the whole ingestion pipeline inside your own perimeter. That last point matters if your documents are regulated, classified or simply too sensitive for a third-party API.

TL;DR

Start with sentence-preserving chunks of about 150 to 500 tokens for precise Q&A. Go larger only for analytical questions, and expect returns to fall off past roughly 2,500 tokens.
Use a small overlap, commonly 10 to 20% of chunk size, to protect sentences at the boundaries. It is a safety net, not a fix for bad structure.
For PDFs, tables and technical manuals, parse layout first (headings, table rows and columns, page headers), then chunk by structure instead of by token count.
Treat chunking as a setting you tune against evaluation metrics, not a constant you pick once.
In air-gapped or regulated environments, parsing, chunking, embedding and vector storage must all run locally, and retrieval must respect user permissions.

What chunking does and why it decides answer quality
Choosing a chunk size
Chunk overlap: how much and why
Chunking strategies compared
Chunking PDFs, tables and technical manuals
Keeping chunking inside your perimeter
Our perspective: chunking is an operations problem
Common mistakes to avoid
Running this in your own environment
FAQ
Sources

What chunking does and why it decides answer quality

A language model cannot read your whole document library at query time. A RAG system embeds small units of text, finds the ones closest to the question, and hands those to the model. The unit you choose is the chunk, so it sets the ceiling on what the model can ever see.

Two failure modes follow directly:

Chunks too small or cut in the wrong place. The retrieved text lacks the context that makes it meaningful. A sentence that says “the limit above does not apply” is useless without the limit.
Chunks too large or mixed. One chunk holds several topics, so its embedding becomes a blurry average and the right passage ranks poorly. The model also receives more noise to sift through.

NVIDIA’s technical blog on chunking strategy puts the cost plainly: poor chunking produces irrelevant results, wasted compute and less business value. The Azure Architecture Center’s chunking guidance frames the goal the same way: chunks that are the right size and each contain semantically relevant content.

Choosing a chunk size

There is no universal number. There is a sensible range per task, and a method for finding yours.

Starting ranges by question type

Starting ranges by question type — overview diagram

Research from early 2026, a systematic analysis of chunking for question answering, reports that fixed-size chunks such as 512 tokens are a common starting point but often struggle with complex enterprise data. It also describes a context cliff: once the context grows past roughly 2,500 tokens, retrieval effectiveness tends to plateau or decline because of dilution and redundancy.

NVIDIA’s chunking write-up reaches a compatible conclusion: smaller, focused chunks work better for factual, semantic Q&A, and larger ones only pay off for analytical questions that need broader context. Public guides differ on exact numbers. One widely read example recommends 512 to 1,024 tokens with 10 to 20% overlap for general document Q&A, and larger windows for long technical papers. Treat these as starting points to test, not answers.

Question type	Starting chunk size	Notes
Short factual lookups (“What torque does step 4 require?”)	150 to 300 tokens	Precision matters most. Keep sentences whole.
General document Q&A	300 to 500 tokens	A sensible default for policies, procedures and reports.
Analytical or multi-part questions	Larger windows, up to about 2,500 tokens	Beyond that, expect dilution. Consider retrieving several smaller chunks instead.
Tables and forms	Structure decides, not tokens	Keep a table, or a coherent group of rows, together with its header.

These ranges come from the sources above and from how retrieval behaves in practice. Your documents may sit outside them.

How to find your own number

Collect 50 to 100 real questions from the people who will use the system, with the passage that answers each.
Build the index at two or three chunk sizes, for example 200, 400 and 800 tokens.
Measure whether the right passage appears in the top results, and whether the generated answer is faithful to it.
Pick the best performer, then repeat when the document set or the question mix changes.

The Azure guidance describes the same loop: gather test documents and queries first, then chunk. The arXiv analysis adds that production teams re-evaluate chunking with RAGAS-style metrics when operational performance drifts. In their framing, chunking is a configuration, not a constant. We cover how to build that measurement loop in our piece on practical retrieval evaluation.

Chunk overlap: how much and why

Overlap repeats the end of one chunk at the start of the next. It protects ideas that straddle a boundary, such as a definition in one sentence and its exception in the next.

A common rule of thumb is 10 to 20% of the chunk size. For a 400-token chunk, that is 40 to 80 tokens. Three cautions:

More overlap is not safer. It inflates your index, returns near-duplicate results and eats the context window you wanted for distinct evidence.
Overlap cannot rescue a bad split. If you cut through the middle of a table, repeating 50 tokens will not restore the missing columns.
Sentence-aware splitting reduces the need for it. NVIDIA’s guidance recommends starting with sentence-preserving methods because they keep topics coherent better than raw token or character counts. With clean sentence boundaries you can often use a smaller overlap.

If you see answers that are right in substance but miss a qualifier, increase overlap slightly or move to structure-based chunks. If you see repeated, near-identical results, reduce it.

Chunking strategies compared

Strategy	How it splits	Best for	Weakness
Fixed-size	Every N tokens or characters	Quick prototypes, uniform prose	Cuts mid-sentence and mid-table
Recursive	Paragraph, then sentence, then word, until under the limit	Mixed text with decent formatting	Ignores meaning and layout
Sentence-based	Groups whole sentences up to a limit	Policies, FAQs, narrative text	Can split a section from its heading
Semantic	Breaks where the topic shifts, based on embedding similarity	Long prose with clear topic changes	Costs compute at ingestion, and needs tuning
Layout or structure-aware	Follows headings, tables, lists and page elements	PDFs, manuals, filings, reports	Needs a good parser
Hierarchical or context-bubble	Retrieves a small chunk, then pulls in its parent section or related content	Complex enterprise libraries	More moving parts to evaluate

A practical order of operations: begin with sentence-preserving or recursive chunking as a baseline, measure it, and move up the table only where the failures justify the complexity. A Stack Overflow-style tip that holds up is to use a basic method first and refine it afterward, since the first pass does not need to be perfect.

Chunking PDFs, tables and technical manuals

This is where most enterprise pipelines struggle. A PDF is a page-drawing format, not a document model. Reading order, headers, footers, columns and tables all have to be reconstructed before any splitting makes sense.

PDF layout elements reconstructed before chunking

Parse layout before you chunk

Microsoft’s documentation on data parsing for agentic retrieval describes capturing structure, tables and images so that search is richer and more accurate. Its document layout approach in Azure AI Search captures headings and chunks the body based on semantic structure rather than arbitrary counts.

In practice, that means:

Detect reading order and drop repeating page headers and footers, which otherwise pollute every embedding.
Keep each table with its header row, and keep row and column relationships intact.
Attach metadata to every chunk: document title, section path, page number, table index and revision date.
Keep captions and the sentence that refers to a figure or table close to it.

Why structure pays off

One 2025 enterprise RAG study found that a hybrid approach preserving table row-column relationships and document hierarchy beat baseline methods, with up to a 16% gain in mean reciprocal rank (0.85 versus 0.69) and faithfulness scores of 4.6 versus 3.0 on a five-point scale. These are results on that study’s proprietary enterprise dataset, so don’t expect the same numbers on yours. The direction is consistent with what practitioners see: structure carries meaning that token counts destroy.

Research on context bubble construction for enterprise retrieval makes a related point. Ranking and selecting top-k passages can fragment document structure, so bundling related, structure-linked content works better than returning isolated slices.

Traceability comes free with metadata

When every chunk knows its source file, section and page, every answer can cite where it came from. For maintenance manuals, compliance procedures and engineering libraries, a source-backed answer is the difference between a tool people trust and one they ignore. It is also what an auditor will ask to see.

Keeping chunking inside your perimeter

Chunking is not a harmless preprocessing step from a security point of view. The raw text of your documents passes through the parser, the splitter and the embedding model. If any of those is a third-party API, the content leaves your environment.

For air-gapped and high-security settings, all of it has to run locally. A 2025 paper on on-premise agentic RAG describes a document processing framework that uses open-source tools to ingest, chunk and embed internal documents into a local ChromaDB vector store, keeping processing within the organization’s infrastructure.

A checklist for a sovereign ingestion pipeline:

Local parsing. OCR and layout detection models run on your hardware, with no cloud parsing calls.
Local embeddings. The embedding model is hosted in-house, and its weights are verified before they enter the network.
Local vector store. Indexes, metadata and source files stay in storage you control.
Offline updates. New documents and model versions arrive through a controlled transfer process, not an open connection. Our guide to verifying artifacts in air-gapped Kubernetes covers that step.
Audit trail. You can show which documents were chunked, with which settings, and when. Platforms built around integrated audit trails and compliance-ready governance show how this kind of record-keeping is handled for large content libraries.

For a wider view of the controls around the pipeline, see our RAG security roadmap and the walkthrough on private RAG for regulated teams.

Chunks and permissions

Good retrieval is not the same as authorized retrieval. Commentary from AI governance specialists, including NHIMG, warns that retrieval quality is a security vector, and that in over-privileged systems the incident rate for unauthorized exposure can reach 76%. The lesson for chunking is to store access metadata on every chunk, such as classification, owning group and document ID, and to filter on it at query time, before ranking. A chunk the user cannot open in the source system should never be eligible for retrieval. The same thinking applies when agents act on retrieved content, which is why context and governance for autonomous agents is worth reading alongside your retrieval design.

The 2025 enterprise study mentioned earlier also reports that combining metadata-aware filtering with cross-encoder reranking raised Precision@5 to 90%, against 75% for baseline RAG, on its own dataset. Metadata does double duty: it improves relevance and enforces policy.

Our perspective: chunking is an operations problem

Most chunking advice is written for a prototype. A notebook, a few PDFs, a public model. The questions change when the corpus is ten thousand controlled documents, the network has no internet and someone has to sign off on every component.

At Forge AI Deployment, we deploy and operate private AI inside infrastructure our customers control, as an official webAI systems integrator. What we see repeatedly is that teams tune the model and neglect the ingestion. Then the first real question, about a table on page 212 of a revision-controlled manual, returns the wrong paragraph with great confidence.

Our view, shaped by leadership experience of more than 20 years in high-consequence and Fortune 100 environments:

Start from the documents, not the framework. Sample the worst files first: scanned pages, nested tables, appendices. If the pipeline handles those, the rest is easy.
Make every answer traceable. Source-backed answers are the point of a technical library assistant, and chunk metadata is what makes them possible. This is the idea behind webAI Frontline, which gives frontline teams fast, source-backed answers from large technical document libraries, fully offline on a single device.
Never trade control for convenience. If parsing or embedding calls leave the perimeter, the data has left the perimeter, whatever the architecture diagram says.
Budget for re-tuning. Document sets drift, and so should chunking settings. Operations, not a one-off build, keeps retrieval accurate.

The pipeline is only one layer. The model, the runtime and the policy enforcement around it need the same discipline, and we cover those in our notes on NIST-aligned private LLM deployment.

— John Ezzell, Founder

Common mistakes to avoid

Picking one size for everything. A policy PDF, a parts catalog and a spreadsheet export need different handling. Tune per document type.
Ignoring tables. Flattening a table to text and then splitting it destroys the very relationships users ask about.
Skipping evaluation. Without a question set and metrics, you are guessing. Faithfulness and retrieval hit rate are a good minimum.
Embedding headers and footers. Repeating boilerplate crowds out real content in similarity search.
Chunking without metadata. You lose citations, filtering and permission checks in one move.
Assuming chunking fixes the wrong architecture. If the real need is changing model behavior rather than grounding it in documents, compare approaches first. Our note on RAG versus fine-tuning lists the operational signals.

Running this in your own environment

If your documents cannot leave your network, the chunking decisions above need to be built into a deployment that is private end to end. Forge AI Deployment designs, deploys and operates sovereign AI systems in enterprise data centers, edge sites and air-gapped networks, using webAI’s local-first platform, including AI Personas, the Intelligence Delivery Network and webAI Frontline. Our engagements cover architecture, secure deployment, model integration and ongoing operation. You can see the full scope on our solutions page, or read about the webAI partnership behind our deployments.

FAQ

What is the best chunk size for RAG?

There is no single best size. For precise question answering, start with sentence-preserving chunks of about 150 to 500 tokens. For analytical questions, larger windows can help, but research points to diminishing returns past roughly 2,500 tokens. Test two or three sizes against real questions from your users.

How much chunk overlap should I use?

A common starting point is 10 to 20% of the chunk size, so 40 to 80 tokens on a 400-token chunk. Increase it if answers lose qualifiers at boundaries, and decrease it if results are repetitive. Structure-aware splitting often lets you use less.

How do I chunk a PDF for RAG?

Parse the layout first: detect reading order, remove repeated headers and footers, and keep tables with their header rows. Then split by headings and sections, and attach metadata such as page and section path to every chunk. Chunking by raw token count alone is what breaks most PDF pipelines.

Can chunking and embedding run fully offline?

Yes. Open-source parsers, local embedding models and local vector stores such as ChromaDB or FAISS can handle ingestion without any external call. In an air-gapped network, model files and document updates must also enter through a controlled, verified transfer process.