RAG Ingestion and Chunking for Enterprise Documents

Most RAG quality problems are ingestion problems wearing a retrieval costume. The pipeline that parses, cleans, chunks, embeds, and indexes your documents sets a ceiling that no downstream cleverness — rerankers, agentic loops, advanced retrieval patterns — can raise. If the right passage was never chunked coherently, tagged correctly, or indexed at all, it cannot be retrieved. This guide covers that unglamorous half of RAG: the pipeline itself.

This is the pipeline layer beneath three companion pieces. Whether agentic RAG is worth building is covered at /guides/agentic-rag-enterprise-guide; how to engineer the agentic loop is at /guides/building-agentic-rag-in-production; and the research-grade retrieval patterns (Self-RAG, RAPTOR, late interaction) are compared at /guides/advanced-rag-patterns-guide. All three assume a clean, well-chunked, deduplicated, well-tagged index. This piece is about earning that assumption.

1. By the numbers

2. The pipeline: parse → clean → chunk → embed → index

Every managed RAG platform has converged on the same stage list, which tells you the shape of the problem is settled even if the parameters are not. Google's Vertex AI RAG Engine describes its process as data ingestion, data transformation ("data is split into chunks"), embedding, data indexing, retrieval, and generation.^[3] Amazon Bedrock Knowledge Bases describes ingestion the same way: documents are parsed, chunked, converted to embeddings, and indexed into a vector store while "maintaining a mapping to the original document."^[4] The stages are stable; the engineering value is in how each one handles enterprise mess.

The first mile is connectors. Bedrock Knowledge Bases, for example, ships connectors for Amazon S3, Confluence, Microsoft SharePoint, Salesforce, a web crawler, and custom data sources — with multimodal content (images, audio, video) supported only through S3 and custom sources.^[5] The connector decision is quietly a metadata decision: what survives into the index — permissions, authorship, timestamps, document type — is bounded by what the connector extracts from the source system. A connector that drops ACL information forces you to rebuild permission awareness later, at much higher cost.

Failure handling has to be designed per document, not per job. A 10-million-document sync will always contain corrupt PDFs, oversize files, and password-protected attachments; one bad file must not poison the batch. Managed platforms model this explicitly — Bedrock exposes per-job statistics and warnings so you can see which documents failed ingestion and why, and enforces file-size quotas per ingestion job.^[6] A self-built pipeline needs the same anatomy: per-document try/catch with a quarantine queue, job-level statistics, and an operator view of what is missing from the index — because a silently absent document is a retrieval failure users will attribute to the model.

3. Chunking: the decision that sets your quality ceiling

Chunking exists for two reasons. The hard one is model limits: Azure's documentation notes the maximum input for the text-embedding-3-small model is 8,191 tokens — roughly 6,000 words — so anything larger must be split.^[7] The subtle one is representation quality: a single vector for a page covering many subtopics represents none of them well, so even content under the limit often retrieves better at finer grain.^[7] Google's guidance states the tradeoff plainly: "A smaller chunk size means the embeddings are more precise. A larger chunk size means that the embeddings might be more general but might miss specific details."^[8]

Fixed-size chunking with overlap is the baseline: a token budget per chunk plus a sliding overlap so sentences straddling a boundary appear in both neighbors. The instructive fact is that the major platforms disagree on the starting point. Bedrock's default chunking splits content into chunks of approximately 300 tokens while honoring sentence boundaries.^[4] Azure recommends starting at 512 tokens (about 2,000 characters) with a 25% overlap of 128 tokens.^[7] Vertex AI RAG Engine defaults to 1,024-token chunks with a 256-token overlap.^[8] A 3.4x spread across three first-tier vendors is the clearest possible signal that defaults are starting points for measurement, not answers.

Structural and hierarchical chunking uses the document's own organization instead of a fixed ruler. Azure's variable-size approach splits on sentence boundaries, markup headings, or detected document structure, and its Content Understanding skill produces semantic chunks that preserve context across page boundaries.^[7] Bedrock's hierarchical chunking goes further: you define parent and child chunk sizes, retrieval matches against the precise child chunks, and the system then "replaces them with broader parent chunks so as to provide the model with more comprehensive context."^[4] That parent-child pattern resolves the precision-versus-context tension directly, at the cost of a more complex index. Boundary discipline matters here: Bedrock's chunker for parsed content "respects logical document boundaries (such as pages or sections) and does not merge content across these boundaries" even when the token budget would allow it^[4] — a rule worth copying in any custom pipeline, because a chunk that welds the end of one section to the start of another embeds a topic that does not exist.

Semantic chunking draws boundaries where meaning shifts rather than where a counter runs out. Bedrock's implementation exposes three hyperparameters — a maximum token count, a buffer size (how many surrounding sentences are embedded together when judging a boundary), and a breakpoint percentile threshold, where a higher threshold requires sentences to be more dissimilar before splitting and so yields fewer, larger chunks.^[4] The documentation is explicit that semantic chunking incurs additional cost because it invokes a foundation model during ingestion.^[4] It also produces variable-length chunks, which complicates capacity planning. For well-formatted documents, structure-based chunking often captures the same boundaries for free; semantic chunking earns its cost on messy, heading-free prose.

Contextual enrichment is the newest addition to the ingestion toolbox: instead of changing where you cut, you change what you embed. Anthropic's contextual retrieval prepends a short, LLM-generated, chunk-specific explanation of how each chunk fits its source document before embedding and indexing it. In their published evaluation — assuming 800-token chunks — contextual embeddings alone reduced the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%); combined with contextual BM25 the reduction was 49% (5.7% → 2.9%); and with reranking added, 67% (5.7% → 1.9%).^[1] The stated one-time cost to generate contextualized chunks was $1.02 per million document tokens.^[1] The strategic point: some of the largest retrieval gains available today are bought at ingestion time, per document, once — not at query time, per request, forever.

4. Deduplication: the noise you ingest is the noise you retrieve

Enterprise corpora are duplicate-rich by construction: the same policy PDF uploaded to four SharePoint sites, near-identical yearly revisions of the same handbook, email threads quoting entire earlier messages, and template boilerplate repeated across thousands of contracts. The best-measured evidence of how duplicated real text collections get comes from LLM training data: Lee et al. found a single 61-word English sentence repeated more than 60,000 times in the C4 web corpus, and that over 1% of the unprompted output of models trained on such datasets is copied verbatim from training data — with deduplication cutting emitted memorized text by roughly ten times.^[2] The mechanism differs in RAG, but the lesson transfers: duplication in the source silently degrades the system built on top of it.

In a RAG index the damage is concrete. Duplicate chunks waste embedding spend and index capacity. Worse, at query time near-identical chunks crowd the top-k: if five of your ten retrieved chunks are variants of the same paragraph, the context window carries one fact five times and starves the answer of coverage — and if the variants disagree (an old policy revision beside a new one), the model is handed a contradiction to resolve on its own. Stuffing a larger context does not rescue this: language model performance "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."^[9]

The practical design: hash-based exact dedup on every document and chunk as a free first pass; near-duplicate clustering on embeddings or shingles as a second pass, with a canonicalization rule that picks the authoritative copy (newest version from the system of record, decided by metadata, not by chance); and a retrieval-time diversity step that suppresses chunks too similar to ones already selected. Versioning deserves special care — an old and new revision of the same document are near-duplicates to a similarity metric but are semantically different in exactly the way that matters, so version-aware canonicalization (supersede, don't merely dedupe) is what prevents stale-answer incidents.

5. Embedding refresh: incremental by default, full recompute on model change

Corpora move. The refresh question is what fraction of the index you touch when they do, and the industry answer for routine change is now settled: incremental. Bedrock's documentation states it directly: "Syncing is incremental, so Amazon Bedrock only processes added, modified, or deleted documents since the last sync." Unchanged documents are skipped; changed documents are re-parsed, re-chunked, re-embedded, and re-indexed; deleted documents are removed from the vector store.^[6] Reproducing that behavior in a custom pipeline requires exactly the idempotency machinery described above — stable identity plus content hashing — which is why idempotency and freshness are the same investment.

A refinement worth copying: when only the filter plane changes, don't re-embed. Bedrock applies a metadata-only optimization that merges updated metadata into existing vectors and writes them back, avoiding calls to the embedding model entirely when content is untouched.^[6] Reclassifying ten thousand documents from "internal" to "confidential" should be a metadata write, not an embedding bill. The exception that proves the design: Bedrock always re-ingests CSV files on metadata change, because their metadata can alter which columns are indexed — content and configuration are entangled there.^[6] Know which of your document types have that entanglement.

Full recompute stops being optional the day you change embedding models. Vectors from different models occupy different spaces; a query embedded with the new model cannot be meaningfully compared against chunks embedded with the old one, so the index must be rebuilt wholesale. The safe pattern is blue/green: build the new index alongside the old, run your retrieval evaluation suite against both, and cut over atomically — never mix model generations in one searchable index. This is also the honest budgeting frame: a model migration costs approximately what the initial build cost in embedding compute, plus any per-chunk enrichment you have added, so "we'll upgrade the embedding model when a better one ships" is a capacity-planning commitment, not a config change.

6. Metadata: the filter plane that makes enterprise RAG governable

Metadata is what separates a semantic search demo from an enterprise system, because it is the layer where tenancy, permissions, recency, and jurisdiction are enforced. The schema must be designed at ingest, since most of it cannot be reconstructed later at acceptable cost: source system and document ID, version and timestamps, owner, access-control tags, document type, and page or section location. The mechanics are mundane and consequential — Bedrock, for instance, pairs each source file with a sidecar metadata file sharing its name,^[6] and choosing the no-chunking option costs you the page-number metadata field, which means losing both page-level citations and page-based filtering.^[4] Chunking and metadata decisions are coupled; make them together.

The filtering architecture question is where the metadata predicate runs. Pre-filtering constrains the candidate set before vector scoring: latency is predictable, cost falls with the candidate pool, and — decisively for governance — out-of-scope content never enters scoring at all. Its failure mode is recall collapse when metadata is sparse or wrong, because a mis-tagged document is invisible no matter how relevant its vectors are. Post-filtering searches broadly and culls afterward: more forgiving of messy metadata, but you pay to score candidates you then discard, and a strict filter can empty the top-k entirely. The production answer is usually hybrid: hard pre-filters for the predicates that are governance constraints (tenant, permissions, region), softer post-filters for preferences (recency, document type), tuned by measurement.

7. Honest objections

"Long-context models make chunking obsolete." The strongest version of this argument is real: context windows have grown enormously, and for small, static corpora, shipping whole documents — or skipping retrieval entirely — is simpler and can be better. But at enterprise corpus scale the economics do not close: you would pay to process the same millions of tokens on every query, and the research record shows models use the middle of long contexts poorly — performance "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."^[9] Chunks are also the unit of citation, of permission filtering, and of freshness tracking; abandoning them abandons the governance surface, not just a token optimization.

"We bought a managed platform, so this is someone else's problem." Partly true — and the true part is worth buying. But the platforms expose chunk size, overlap, hierarchy depth, and semantic thresholds as knobs precisely because they cannot choose for you, and their own defaults disagree by more than a factor of three (300 vs. 512 vs. 1,024 tokens).^[4][7][8] Metadata schema design, canonicalization rules, refresh cadence, and retrieval evaluation are not delegable at all. A managed platform moves your effort up the stack; it does not remove it.

"Just use semantic chunking for everything." Semantic chunking is genuinely better on unstructured, heading-free prose. But it costs foundation-model invocations at ingest,^[4] produces variable-size chunks that complicate operations, and for the well-formatted documents that dominate many enterprise corpora — policies, manuals, contracts — the structure already encodes the semantics, so structural chunking captures the same boundaries at near-zero cost. The honest default is structural-first with measurement, not semantic-everywhere on principle.

8. The read

For a platform lead sequencing investment, the order is: pipeline correctness first (idempotent upserts, delete propagation, per-document failure handling), metadata schema second (permissions and provenance captured at ingest, enforced as pre-filters), chunking third (structural with a fixed-size fallback, starting inside the vendor-default band of roughly 300–1,000 tokens and 10–25% overlap,^[4][7][8] then tuned against a golden-query suite), deduplication fourth, and enrichment last — contextual retrieval's 49–67% failure-rate reductions^[1] are the reward for a pipeline mature enough to re-process its corpus on demand.

The portable takeaway: treat the ingestion pipeline as a versioned data product with a schema, an SLA, and an evaluation suite — not as a preprocessing script that ran once. Every advanced pattern in the companion guides inherits its ceiling from this layer. Teams that can re-chunk, re-embed, and re-index their corpus safely in a day can adopt every future improvement in retrieval; teams that cannot are frozen at the quality of their first ingestion run.