Building Agentic RAG in Production: Migration, Query Planning, Tools, Latency, and Evaluation

The engineering path from a working standard-RAG pipeline to an agentic retrieval loop runs through five decisions: what to keep from the existing stack, how the model plans and routes queries, how tools are described, where latency is capped, and how the loop is evaluated. Get tool descriptions and the loop budget right and most of the rest follows.

1. What actually changes when retrieval becomes a loop

The Agentic RAG survey frames the limitation you are engineering around: traditional RAG systems are "constrained by static workflows and lack the adaptability required for multi-step reasoning and complex task management." Agentic RAG addresses that by "embedding autonomous AI agents into the RAG pipeline" — agents that "dynamically manage retrieval strategies, iteratively refine contextual understanding, and adapt workflows."^[2]

Anthropic's distinction between workflows and agents is the practical dividing line for your architecture. Workflows are "systems where LLMs and tools are orchestrated through predefined code paths"; agents are "systems where LLMs dynamically direct their own processes and tool usage."^[5] A standard RAG chain is a workflow: retrieve once, then generate. An agentic RAG system hands the model one or more retrieval tools and lets it decide what to search, judge what came back, and search again — the pattern ReAct established by interleaving reasoning traces with actions against external knowledge sources.^[1]

2. Migration: keep the data layer, replace the orchestration

The lowest-risk migration sequence has four steps, and the first two produce no user-visible change. First, modularize: put a clean interface in front of your retriever — query in, ranked passages out — so it can be exposed as a tool. If retrieval logic is tangled into prompt assembly, this is where you untangle it. Second, wrap and baseline: register the existing retriever as the loop's only tool and cap the loop at one iteration. Behavior should be functionally identical to the old chain, which gives you a regression baseline for quality, latency, and cost before any agentic behavior exists.

Third, let the loop iterate — raise the cap, allow query reformulation, and watch the trajectories. Only after single-source iteration is stable should you, fourth, add sources and tools: a second knowledge base, a structured database, an internal API. Each addition changes the routing problem the model must solve, so add them one at a time with the evaluation suite running. Capture baseline telemetry before every step, because the loop will be slower than the chain and you need to know by how much and where. Microsoft says this plainly about its own managed implementation: "Agentic retrieval adds latency compared to a single-query pipeline, but it handles query complexity that a single query can't."^[6]

3. Query planning: decomposition, routing, and joins

Once the loop is live, answer quality on hard questions is decided by three planning sub-problems. They are worth engineering as distinct components, because they fail in distinct ways.

This layer is now productized, and the managed implementations are useful reference architectures even if you build your own. Azure AI Search's agentic retrieval is a multi-query pipeline in which an LLM breaks a complex query "into smaller, focused subqueries" that can include chat history for context; all subqueries "run simultaneously," each is semantically reranked, and the system "combines the best results into a unified response" with source references and an activity log.^[6] Amazon's managed Bedrock Knowledge Bases describe the same shape: agentic retrieval that "supports multi-hop reasoning, decomposes complex queries into sub-queries, retrieves iteratively across multiple knowledge bases, and evaluates sufficiency of responses."^[7]

Two engineering rules fall out of those designs. First, planner effort should be tunable per query: Azure exposes retrieval reasoning-effort levels, and at the minimal level the LLM planning step is skipped entirely and queries go straight to the knowledge sources.^[6] Build the same escape hatch in-house — classify the query cheaply first, and spend planning tokens only on compound questions. Second, treat joins as a ranking problem, not an assembly problem: rerank the merged candidate pool and truncate hard, and when two sources genuinely conflict, surface the conflict to the synthesis step rather than letting one source silently win.

4. Tool design: the description is the interface

In an agentic RAG system the model selects tools by reading their definitions, so schema and description quality is the highest-leverage line of code you will write. Anthropic's docs state the mechanism directly: Claude "determines when to call a tool based on the user's request and the tool's description," with each tool defined by a name, description, and JSON input schema.^[8] OpenAI's function-calling guide gives the same instruction from the other side: "Write clear and detailed function names, parameter descriptions, and instructions" and explicitly describe what each parameter's format and output represent.^[3]

• Make invalid calls unrepresentable. Use enums, required fields, and typed parameters so the schema itself rules out bad states — OpenAI's guide names the anti-pattern: a toggle_light(on: bool, off: bool) signature that permits contradictory calls.^[3] Anthropic offers strict tool use to guarantee calls conform to the declared schema.^[8]
• Keep the exposed toolset small and consolidated. OpenAI suggests staying under roughly 20 functions per turn as a soft ceiling.^[3] Anthropic recommends consolidating overlapping operations — one well-designed scheduling tool instead of separate list_users, list_events, and create_event tools — and namespacing related tools under common prefixes when the count grows.^[9]
• Return high-signal, token-bounded results. Anthropic's guidance: tool implementations "should take care to return only high signal information back to agents," with pagination, filtering, and truncation behind sensible defaults.^[9] A retrieval tool that dumps 40 raw chunks per call fills the context window the loop needs for its next decision.
• Wrap databases at the intent level. Expose parameterized operations (find_customer_orders(customer_id, since)) rather than a raw SQL string parameter. The schema then enforces least privilege, and query mistakes become schema-validation errors instead of injection surface.
• Treat permissions as part of the connector. Amazon's managed Knowledge Bases ship connectors for S3, SharePoint, Confluence, Google Drive, and OneDrive with document-level permission filtering via access control lists applied at retrieval time.^[7] That is the bar: if you build connectors in-house, permission-aware retrieval is your problem, and an agent that retrieves across sources will happily join data its caller should never have seen together.

Tool results are the agent's ground truth. Anthropic's agent-design guidance emphasizes that agents must gain "'ground truth' from the environment at each step" — tool call results, execution output — to assess progress.^[5] That makes error messages part of the interface: a tool that returns an actionable error ("date must be ISO 8601; got '3/4/25'") lets the loop self-correct in one step, while an opaque failure burns an iteration of your loop budget and often derails the plan.

5. Latency engineering: budget the loop, then spend it

An agentic loop spends latency in multiples: every iteration is at least one model call plus one tool round trip, and the context grows as trajectories lengthen. The workable posture is a per-query budget — iterations, tokens, wall-clock — set before the first production request, with the levers below tuned inside it.

Token discipline is latency work by another name. Tool definitions are billed and processed as input tokens on every request that carries them, and Anthropic's docs note the tool-use system prompt itself adds a few hundred tokens on top.^[8] Verbose tool results compound worse, because they ride along in context for every subsequent iteration — which is exactly why Anthropic's tool-writing guidance pushes pagination, filtering, and truncation with sensible defaults.^[9] The cheapest token is the one the loop never has to process again.

6. Evaluation: correctness, efficiency, and tool-use accuracy

Standard RAG evaluation carries over as the first dimension. The RAGAS framework evaluates a RAG pipeline "without having to rely on ground truth human annotations," scoring the retrieval system's ability to find relevant, focused context passages, the LLM's ability to use those passages faithfully, and the quality of the generation itself.^[10] Those reference-free metrics remain useful per retrieval pass — but they say nothing about whether the agent took a sensible path to get there.

Agentic systems add two dimensions the pipeline world never needed. Tool-use accuracy asks whether the agent selected the right tool, passed schema-valid and semantically correct arguments, and interpreted the result correctly — failures here cascade invisibly, because a well-written answer grounded in the wrong lookup passes every text-quality check. Efficiency asks what the trajectory cost: Anthropic's tool-evaluation guidance is to collect metrics beyond accuracy, "including runtime, tool call counts, and token consumption."^[9] The Agentic RAG survey's taxonomy — agent cardinality, control structure, autonomy, and knowledge representation — is a useful set of axes when you are comparing candidate architectures against the same evaluation suite.^[2]

Operationally, run evaluation at two levels. Offline, a hold-out task suite acts as the regression gate for every planner, prompt, or tool change — and in an agentic system, a reworded tool description is a behavior change and should be treated like a code deploy. Online, sample production trajectories continuously; loop systems drift as source content, tool latencies, and query mixes shift, and the trajectory log is where tool-selection regressions show up first.

7. The framework and platform landscape

The build-vs-buy line in agentic RAG now sits at the query-planning layer. Below it, open-source frameworks give you the loop and the data plumbing: LangGraph positions itself as a low-level orchestration framework for stateful agents, with durable execution and human-in-the-loop state inspection^[11], while LlamaIndex frames itself as an open-source framework for agentic applications, supplying data connectors for APIs, PDFs, documents, and SQL plus the retrieval interface agents call as a tool^[12]. Above the line, the hyperscalers are absorbing planning into managed retrieval services — including Google's Vertex AI RAG Engine, a managed framework spanning ingestion, transformation, embedding, indexing, retrieval, and generation^[13].

The strategic read on this table: if you build the planner in-house, the managed pipelines define the behavior bar your version will be compared against — parallel subqueries, semantic reranking, tunable reasoning effort, citations, and an activity log. If you buy, the engineering that remains is exactly this guide's other four sections: migration sequencing, tool design, loop budgets, and evaluation. No vendor manages those for you, and they are where production quality is actually decided.

8. Honest objections

"Most of our traffic doesn't need this." Usually true, and the vendors selling agentic retrieval concede the cost — Microsoft's own documentation states that agentic retrieval adds latency over a single-query pipeline.^[6] A well-tuned single-pass pipeline with query rewriting and a reranker covers a large share of enterprise question-answering, and Anthropic's guidance to add complexity only when it demonstrably improves outcomes^[5] is an argument for restraint, not adoption. The honest response is routing: hold the single-pass path for single-hop queries and reserve the loop for the query classes where single-pass measurably fails. If you cannot find those failing classes in your logs, you do not need agentic RAG yet.

"Nondeterminism is an operational tax we can't price." Also fair. The same input can produce different trajectories; debugging means reading tool-call transcripts rather than stack traces; and every prompt or tool change needs an evaluation run that costs real tokens. This is why trajectory logging and the eval suite appear inside the migration path in this guide rather than as afterthoughts — they are the price of admission. An organization that will not fund continuous evaluation should stay with a deterministic workflow, because an unevaluated agentic system does not fail loudly; it regresses silently.

9. The read

Five decisions travel with you regardless of stack. Keep the data layer and replace only the orchestration, entering through a capped loop that reproduces your current chain. Treat tool schemas and descriptions as the interface with the most leverage per engineering hour. Give the loop an explicit budget — iterations, tokens, wall-clock — and spend it with parallelism and caching. Evaluate trajectories, not just answers, and gate every prompt and tool change on the suite. And place your build-vs-buy line consciously at the query-planning layer, knowing the managed services are moving it up every quarter.