Multimodal AI in the Enterprise: Architecture, Video Understanding, and Use Cases

Multimodal AI — models that take images, video, and audio alongside text — is now a line item, not a lab demo. The decisions that matter are architectural: which fusion pattern you are actually buying, what pixels cost in tokens at your volume, and when a unified frontier model beats a specialized perception service. This guide covers all three, plus the video-understanding patterns that are production-ready today.

1. By the numbers

2. The real choice: unified model or specialized perception service

The framing question for a CIO or platform lead is not "which multimodal model is best." It is whether a given workload belongs on a unified multimodal LLM (Claude, GPT-series, Gemini), on a specialized perception service (Amazon Rekognition Video, Azure AI Video Indexer), or on a hybrid where a perception layer extracts structure and an LLM reasons over the extractions. The two families are built differently, priced differently, and fail differently.

3. How vision and language actually get combined

Four research milestones explain essentially every commercial multimodal model you can buy today, and knowing them tells you what you are paying for. The lineage matters because it dictates both capability boundaries and pricing mechanics.

Patches made images speakable. The Vision Transformer (ViT) paper showed in 2020 that "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks," while requiring substantially fewer computational resources to train than state-of-the-art convolutional networks.^[5] Cutting an image into fixed-size patches turns it into a token sequence — the same shape as text. That single idea is why every major API now meters images in tokens rather than per-image flat fees.

Contrastive pretraining aligned the modalities. CLIP demonstrated that "the simple pre-training task of predicting which caption goes with which image" scales into state-of-the-art image representations when trained on 400 million (image, text) pairs, and that natural language can then reference those visual concepts zero-shot — CLIP matched the accuracy of the original ResNet-50 on ImageNet without using any of the 1.28 million labeled training examples ResNet-50 was trained on.^[1] A shared embedding space where an image of an invoice and the word "invoice" land near each other is what makes visual search, cross-modal retrieval, and zero-shot classification work.

Bridging beat retraining. DeepMind's Flamingo showed you can "bridge powerful pretrained vision-only and language-only models," handle "sequences of arbitrarily interleaved visual and textual data," and "seamlessly ingest images or videos as inputs" — with a single model reaching new few-shot state of the art simply by being prompted with task examples.^[6] Rather than training one giant model from scratch, Flamingo froze strong unimodal models and trained lightweight cross-attention connections between them.

The projection pattern made it cheap to replicate. LLaVA connected "a vision encoder and LLM for general-purpose visual and language understanding" and used language-only GPT-4 to generate the multimodal instruction-following data it was tuned on.^[7] A pretrained vision encoder, a thin projection layer mapping visual features into the language model's embedding space, and an instruction-tuned LLM decoder — that recipe is now the default template for open-weight multimodal models, and it is why vision capability can be bolted onto an existing LLM rather than requiring a new foundation model.

The buyer's takeaway: commercial multimodal models are, in spirit, late-fusion systems — a vision encoder feeding a language decoder through an alignment layer. Vision quality and language quality are semi-independent, which is why a model can be excellent at reading a dense table yet mediocre at counting objects, and why you should evaluate the visual tasks you actually run rather than trusting a single "multimodal" benchmark number.

4. What pixels cost: the token economics of images and video

Because images enter the model as patch tokens, multimodal cost is a geometry problem, and every major vendor now publishes the formula. Anthropic documents that Claude views images as 28×28-pixel patches, each one a visual token, so an image costs ⌈width/28⌉ × ⌈height/28⌉ tokens — 1,296 tokens for a 1000×1000-pixel image. Standard-tier models downscale anything past a 1,568-pixel long edge to cap cost at 1,568 tokens per image, while high-resolution models (Claude 4.7 and later) allow a 2,576-pixel long edge and up to 4,784 tokens; Anthropic's own worked example prices a thousand 1-megapixel images at about $1.30 at a $1-per-million-input-token rate.^[2] OpenAI likewise documents patch-based tokenization (32×32-pixel patches on its GPT-5-series models) alongside an older tile-based scheme for the GPT-4o family, with `low`, `high`, `original`, and `auto` detail settings that trade cost against fidelity.^[8]

Video is metered the same way, one sampled frame at a time. The Gemini API stores uploaded video at 1 frame per second and tokenizes each second at roughly 300 tokens at default media resolution — 258 tokens per frame plus 32 tokens per second for the audio track — or about 100 tokens per second at low resolution.^[3] Run the arithmetic before you commit: at the documented default rate, a single hour of footage consumes on the order of a million input tokens, which is exactly why Gemini gates hour-plus videos behind its 1M-context models and offers the low-resolution mode to stretch that to three hours.^[3]

Volume limits shape architecture too. The Claude API accepts up to 100 images per request on 200k-context models and up to 600 on others, at up to 10 MB per image, in JPEG, PNG, GIF, or WebP.^[2] OpenAI documents up to 1,500 individual image inputs per request within a 512 MB payload.^[8] Those ceilings sound generous until you point a pipeline at a document archive or a camera feed — then the governance question becomes a resizing policy: who decides what resolution each workload actually needs, and where in the pipeline images get downsampled before they hit a metered API.

5. Video understanding: frame sampling is the design decision

Every video system — research or commercial — starts by throwing most of the frames away. A camera shoots 30 or 60 frames a second; no economically viable model reads them all. What used to be an internal research knob is now a documented platform parameter: Gemini samples stored video at 1 frame per second, and its `media_resolution` setting controls how many tokens each retained frame consumes.^[3] That 1 FPS number should anchor your expectations. Reading a slide, a whiteboard, or a dashboard that stays on screen for seconds works well. Catching a sub-second event that falls between samples — a dropped item, a fast gesture, a flicker — is precisely what a 1 FPS pipeline can miss, no matter how good the model is.

Meeting intelligence is mostly an audio problem wearing a video costume. Gemini's own accounting makes the point: audio is metered at 32 tokens per second against 258 for each default-resolution frame.^[3] The transcript carries the decisions, owners, and action items; the frames earn their cost only when slides, screen shares, or demos carry information speech doesn't. A cost-rational meeting pipeline transcribes and diarizes everything and applies frame understanding selectively — the speech side of that stack (transcription, diarization, voice interfaces) is the subject of /guides/enterprise-voice-ai-guide.

Camera monitoring is where specialized services still earn their keep. Amazon Rekognition Video analyzes video stored in S3 for labels, faces, people, text, segments, and explicit content through an explicitly asynchronous pattern — you call a Start operation such as StartLabelDetection, completion status is published to an SNS topic (typically consumed via SQS or Lambda), and a Get operation returns timestamped results; input must be H.264-encoded MPEG-4 or MOV, up to 10 GB per stored video, and a separate streaming-video path handles live feeds.^[4] That job-queue shape is what continuous, archive-scale video actually needs, and its structured, timestamped output is what an auditor or an incident review can use.

Azure AI Video Indexer takes the pipeline-of-models approach to its logical end: one ingestion pass runs face detection, OCR, object detection, scene/shot/keyframe segmentation, and label identification on the visual track, plus transcription in more than 50 languages, speaker enumeration for up to 16 speakers, translation, and topic inference on the audio track — and it can run in the cloud or at edge locations via Azure Arc for deployments with data-residency or latency constraints.^[9] For a media library or a compliance archive, that battery of narrow models produces searchable, timestamped metadata that a single LLM prompt cannot match on cost or consistency.

6. The enterprise use-case landscape

Across industries, the workloads that reach production cluster into six families. Treat these as patterns to locate your own candidates in, not as proof that any particular vendor delivered any particular result — the pattern, not the anecdote, is what transfers.

Two screening questions sort candidates quickly. First: is the visual input load-bearing, or decorative? If the text alone answers 95% of queries, you have a text project with an image garnish — price it that way. Second: is the output open-ended or schema-stable? Open-ended reasoning favors a unified model; stable schemas at volume favor a perception service or a fine-tuned narrow model.

7. Honest objections

"We already have OCR and computer vision — why pay token rates?" Often you shouldn't. A barcode scan, a fixed-template extraction, or a single-class detector is cheaper, faster, and more predictable as a purpose-built system. The multimodal premium is justified by variety — unstructured layouts, mixed modalities, questions you couldn't enumerate in advance — not by volume of a repetitive task.

"The accuracy limits are real." They are, and the vendors say so themselves. Anthropic's documentation states that Claude may hallucinate on low-quality, rotated, or very small images, gives approximate rather than exact object counts, produces approximate spatial coordinates, cannot determine whether an image is AI-generated, and is not designed to interpret diagnostic scans such as CTs or MRIs.^[2] Any workflow where a miscount or a misread field carries material cost needs human review or deterministic cross-checks in the loop — treat the model as a drafting layer, not an oracle.

"Video costs will run away from us." At documented per-second token rates this is a legitimate risk for anyone pointing an LLM at raw footage.^[3] The mitigation is architectural: use cheap perception (motion detection, scene segmentation, transcript search) to select the minutes worth deep analysis, and send only those to the expensive model. Nobody should be paying frontier-model rates to watch an empty hallway.

"This locks us deeper into one vendor." Partially true. Prompt-level portability across multimodal APIs is reasonable, but token-accounting formulas, resolution tiers, and file-handling mechanics differ by vendor — a cost model tuned for one platform does not transfer unchanged.^[2][8][3] Keep preprocessing (resizing, sampling, chunking) in your own pipeline code rather than delegating it to vendor defaults, and re-run your cost model per candidate vendor during selection.

8. The read: how to decide

Default to a hybrid architecture for anything involving long video: specialized perception extracts structure — transcripts, segments, detections, timestamps — and a multimodal LLM reasons over the extractions plus a handful of selected frames. Use a unified multimodal model directly when the task is genuinely open-ended visual reasoning over bounded inputs: documents, charts, screenshots, short clips. Use a specialized service alone when the output schema is fixed and the volume is high. And treat token accounting as a first-class selection criterion: the architecture lineage above means every vendor is metering geometry, and the vendor whose resolution tiers match your actual fidelity needs will quietly be the cheapest at scale.