Multimodal Document Intelligence: Processing Documents, Charts, and Tables with LLMs

Most of what an enterprise knows lives in documents that were formatted for human eyes: invoices with line-item tables, contracts with clause hierarchies, filings with embedded charts. Strip those to plain text and you destroy the layout that carried half the meaning. Multimodal document intelligence is the stack decision about how to keep it.

That stack now has three distinct layers, and the platform decision most teams get wrong is treating them as substitutes rather than complements. Deterministic document-AI services (Azure AI Document Intelligence, Amazon Textract, Google Document AI) extract fields and tables with schemas and confidence scores. Frontier LLMs (Claude, Gemini) now ingest PDFs natively, seeing each page as an image and reasoning over charts and diagrams no parser can express. And a new generation of retrieval models embeds page images directly, so RAG systems can find a figure or a table cell without ever perfectly parsing it. This guide maps what each layer verifiably does, what it costs, and how a CIO or platform lead should divide the work — with the adjacent decisions covered in /guides/enterprise-multimodal-ai-guide for the broader multimodal platform question.

1. By the numbers

2. The real decision: parse first, or let the model see the page

For a decade, document automation meant a parse-first pipeline: OCR the page, detect layout regions, extract key-value pairs and tables into a schema, then hand structured data to downstream systems. The cloud doc-AI services industrialized this pattern and remain very good at it. The alternative that matured in the last two years is native document vision: send the PDF to a multimodal LLM and let it read the page the way a person does. Claude's PDF support converts each page into an image, extracts the text, and provides both to the model, which is what lets it answer questions about charts, diagrams, and other non-textual content rather than just transcribed words.^[2] Gemini's document understanding works the same way at larger scale — native vision over documents up to 1,000 pages, with each page normalized to at most 3072x3072 resolution.^[1]

Neither approach dominates. They fail differently, cost differently, and audit differently — which is exactly why the architecture question is a division of labor, not a bake-off. The parse-first pipeline gives you typed fields, confidence scores, and a provenance trail regulators recognize, but it only extracts what its schema anticipated. The native-vision path handles anything a person could read, including the figures, but its errors arrive fluently and without a confidence signal. Mature deployments route by document class: schema-stable volume goes to the deterministic layer, and the long tail — plus everything with a chart in it — goes to the model that can see the page.

3. The structured-extraction layer: what the doc-AI services verifiably do

The three hyperscaler services have converged on a similar shape: a general layout/OCR model, a catalog of prebuilt extractors for common document types, and a way to train custom extractors on your own forms. Azure AI Document Intelligence pairs OCR with document-understanding models that extract text, tables, structure, and key-value pairs, offers a Layout model for text, tables, and document structure, and ships prebuilt models for invoices, receipts, identity documents, tax forms, and contracts — plus custom template models for fixed layouts and custom neural models for structured, semi-structured, and unstructured documents.^[5]

Amazon Textract covers the same ground with a different API decomposition: Document Analysis for text, forms, and tables; a Queries feature that lets you ask for specific fields in natural language (with Custom Queries to tune it on your data); AnalyzeExpense for invoices and receipts; AnalyzeID for US government-issued identity documents; and an Analyze Lending workflow that classifies mortgage-package pages and routes each to the right analysis operation. Synchronous calls handle latency-critical single pages; asynchronous operations handle multipage documents.^[6]

Google Document AI is the most explicitly RAG-aware of the three: alongside OCR, form and table extraction, custom extractors with up-training, and document classification and splitting, its Layout Parser extracts text, tables, and lists and returns context-aware chunks — pre-chunked output designed to feed retrieval pipelines directly.^[7] If your document corpus is heading into a RAG system, that chunking step is where most quality is won or lost; the mechanics are covered in depth in /guides/rag-ingestion-and-chunking.

4. Native document vision: what changes when the model sees the page

The native-vision path removes the pipeline entirely for a large class of tasks. Claude accepts standard PDFs up to 32 MB and 600 pages per request (100 pages when the request's context window is under 1M tokens), and because every page travels as text plus image, you can ask about the pictures: financial-report charts, legal-document structure, visual layout.^[2] Gemini's limits run to 50 MB or 1,000 pages, and its documentation is blunt about the boundary that matters: document vision only meaningfully understands PDFs — TXT, Markdown, or HTML inputs are extracted as pure text, losing charts and diagrams.^[1] That is a useful rule for your own architecture, too: the moment you flatten a document to text upstream, no downstream model can recover what the page looked like.

The cost math deserves a spreadsheet before a rollout. At roughly 258 tokens per page, a 100-page document costs Gemini about 26K input tokens; Claude's text extraction alone runs 1,500–3,000 tokens per page depending on density, with image-token costs on top.^[2] For a workload of millions of pages a month, that is the difference between a rounding error and a line item — and it is why high-volume, fixed-schema extraction usually stays on the per-page-priced deterministic services while the LLM handles the documents that defeat them. Prompt caching and batch processing, both documented for Claude's PDF workflow, are the first levers to pull when the LLM share of the volume grows.^[2]

5. Charts and tables: from pixels to data

Charts are the hardest documents in the building, because the data is encoded geometrically — bar heights, axis scales, legend colors — and answering a real question requires both reading the pixels and doing arithmetic. The ChartQA benchmark was built precisely because earlier datasets leaned on template questions with fixed-vocabulary answers; its 9,600 human-written questions demand the logical and arithmetic operations people actually perform on charts, and its authors found that even state-of-the-art models struggled on the complex reasoning ones.^[4] That result has aged into a design principle: do not ask one opaque step to do perception and reasoning at once.

The DePlot result showed why decomposition wins. By translating a plot into a linearized table first, then letting an LLM reason over the table, DePlot plus few-shot prompting beat a finetuned state-of-the-art model — one trained on more than 28,000 examples — by 24.0% on ChartQA's human-written queries, using just one-shot prompting.^[3] The enterprise translation: when a chart's numbers matter, have your pipeline recover the underlying table (from the source system if you can, from a plot-to-table step if you cannot) and let the language model reason over structured data. Reserve direct chart-reading for triage, summarization, and questions where an approximate read is acceptable.

6. Multimodal RAG: three ways to retrieve what you processed

Extraction is half the system; the other half is finding the right page, table, or figure at question time. Three retrieval architectures are in production use, and they trade preprocessing effort against fidelity.

The first pattern is the pragmatic default because it reuses the text-RAG stack you already run, and services like Document AI's Layout Parser exist to feed it well-formed, context-aware chunks.^[7] The second pattern removes the captioning bottleneck: Anthropic's embeddings documentation (Anthropic offers no embedding model of its own and points to Voyage AI) lists voyage-multimodal-3 as vectorizing interleaved text and content-rich images — screenshots of PDFs, slides, tables, figures — with a 32,000-token context length and 1024-dimension vectors, and voyage-multimodal-3.5 extending coverage to video.^[8] A chart indexed this way can be retrieved by what it shows, not by what a captioner happened to say about it. Model-selection tradeoffs for this layer, including dimensions and cost, are covered in /guides/enterprise-embedding-models-guide.

The third pattern is the research front that went practical. The ColPali authors observed that modern retrieval pipelines exploit documents' visual features poorly — figures, layouts, tables, fonts — and introduced the ViDoRe benchmark for page-level retrieval across domains, languages, and practical settings. Their model embeds page images directly into multi-vector representations and, combined with late-interaction matching, largely outperforms modern document retrieval pipelines while being drastically simpler, faster, and end-to-end trainable.^[9] For corpora where the visuals are the content — technical drawings, financial exhibits, scanned forms — vision-native retrieval removes the single largest source of silent failure: the parse you never knew was wrong. The cost is infrastructure maturity; multi-vector late interaction is not yet a checkbox in every vector database, so pattern 3 is a deliberate platform bet, not a drop-in swap.

7. Operating it: evaluation, drift, and access control

Whatever mix of layers you land on, the operational disciplines are the same, and they are where document-intelligence programs actually succeed or stall. First, evaluation has to be yours. Vendor demos run on clean documents; your corpus has faxed receipts, rotated scans, and the one supplier whose invoices break everything. Build a held-out gold set from your own documents — spanning layouts, quality levels, and the charts and tables that matter — and score extraction with precision and recall per field, not a single blended accuracy number that hides which fields fail. The same set becomes your regression gate for every model version bump, prompt change, and vendor switch; without it, every upgrade is a leap of faith.

Second, plan for drift. Document formats change without notice — a supplier redesigns an invoice template, a regulator revises a form — and both deterministic extractors and LLM prompts degrade quietly when they do. Confidence-score distributions and per-field exception rates are your early-warning instruments on the deterministic layer; sampled human review plays that role for LLM output, which arrives fluent whether or not it is right. Third, retrieval inherits your security model. Page images and figure embeddings are data, and a vector index that returns a chart from a restricted board deck to an unrestricted user is a breach with extra steps — so document-level permissions must be enforced at query time in the retrieval layer, with metadata that ties every chunk, image, and embedding back to its source document and page. That provenance chain is also what makes an extracted answer auditable when someone asks where a number came from.

8. Honest objections

The strongest counterargument to all of this: for most document workloads, boring OCR plus templates is still the right answer, and multimodal LLMs add cost and nondeterminism you do not need. That is frequently true. If 90% of your volume is invoices in twelve known layouts, a prebuilt extractor with confidence thresholds and human-in-the-loop review will beat an LLM pipeline on cost, latency, and auditability — and the vendors' own positioning supports it: the deterministic services exist precisely for high-accuracy, reliable extraction from structured documents.^[5] The LLM layer earns its place on the long tail, not the head, of the layout distribution.

A second objection: chart understanding is still unreliable, so building workflows on it is premature. The benchmark record backs the caution — ChartQA's authors reported persistent obstacles on complex reasoning questions even for state-of-the-art models^[4] — and nothing in the vendor documentation promises numeric fidelity when reading a bar chart. The honest posture is to treat model-read chart values as drafts requiring verification wherever the number is load-bearing, and to engineer around the weakness with plot-to-table decomposition, which is measurably stronger.^[3] Finally, on vision-native retrieval: it is young. ViDoRe dates to 2024,^[9] enterprise tooling is uneven, and a team without multi-vector infrastructure may rationally wait a cycle while running pattern 1 or 2.

9. The read

Buy the deterministic layer, rent the vision layer, and pilot the retrieval layer. The extraction services are mature, priced per page, and auditable — they should own every fixed-schema, regulated, high-volume field. Native document vision is a per-token utility for the documents that defeat schemas: long-tail layouts, cross-page reasoning, figures. And retrieval is where the architecture is genuinely moving — commercial multimodal embedding models now ship as standard offerings — Voyage's, for one, are documented as vectorizing interleaved text and images^[8] — and vision-native late interaction is the direction of travel for visually rich corpora.^[9] The teams that get this right are not choosing a winner; they are drawing the boundary lines deliberately, instrumenting each layer, and re-drawing the lines as the per-token economics shift.