Production ML Pipelines: CI/CD, DAGs, Event-Driven Architectures, and Error Handling

If you own an ML platform, four decisions determine whether it survives contact with production: what your deployable unit is (the pipeline, not the model), how you structure work (a DAG of idempotent steps), when you trigger it (schedule versus event), and what happens when a step fails (a retry policy tied to a failure taxonomy). Everything else is tooling.

Those four decisions are not vendor-specific, but the vendors have written down more operational detail than most internal platform teams ever will. Google Cloud's MLOps maturity model defines what "automated" actually means at each level^[1]; AWS documents retry semantics down to the default backoff multiplier^[2]; Azure spells out why a pipeline of separable steps beats a monolith^[3]. This guide consolidates that material into a decision path a platform lead can defend in an architecture review.

1. By the numbers

2. The framing: three maturity levels, one deployable unit

The most useful mental model in this space remains Google Cloud's three MLOps levels^[1]. At level 0, "every step is manual, including data analysis, data preparation, model training, and validation," and what gets deployed is only "the trained model as a prediction service (for example, a microservice with a REST API), rather than deploying the entire ML system." At level 1, the goal is continuous training: "you deploy a whole training pipeline, which automatically and recurrently runs to serve the trained model as the prediction service." At level 2, CI/CD automates the pipeline's own build-and-release cycle, which "lets your data scientists rapidly explore new ideas around feature engineering, model architecture, and hyperparameters."

The pivot to internalize is the change of deployable unit. Traditional CI/CD ships an application; ML CI/CD at level 1 and above ships a factory. Once the training pipeline is the thing under version control, test coverage, and release management, model refreshes stop being events and become throughput. That single reframing is what separates teams that retrain weekly from teams that renegotiate a quarterly "model update project" every time drift bites.

3. CI/CD for ML: what actually goes in the gate

The reason ML CI/CD is harder than application CI/CD is scope. Google's architecture guide is blunt that "only a small fraction of a real-world ML system is composed of the ML code" — the rest is configuration, data collection, testing, serving infrastructure, and resource management^[1]. A CI gate that only lints and unit-tests the model code is validating the small fraction and waving the rest through.

So the practical CI/CD gate for an ML pipeline has three layers. First, conventional software checks on pipeline code and component containers: unit tests, integration tests, image builds. Second, data validation: schema checks, distribution checks, and freshness checks run as pipeline steps, so bad inputs fail fast before compute is spent on training — the governance side of this is covered in depth in /guides/data-quality-governance-ai. Third, model validation: evaluation against a holdout set with explicit promotion thresholds, so a model that regresses never auto-promotes. The threshold is a product decision encoded in the pipeline, not a judgment call made in a notebook at 6 p.m.

The managed platforms have converged on this shape. Amazon SageMaker Pipelines positions itself as "a purpose-built workflow orchestration service to automate machine learning (ML) development," running on serverless infrastructure that AWS provisions, scales, and shuts down on demand, with built-in versioning and lineage tracking across executions^[7]. Azure Machine Learning defines a pipeline as "a workflow that automates a complete machine learning task," broken into steps that can be developed, automated, and owned separately, with the platform managing inter-step dependencies^[3]. Both designs assume the pipeline — not the notebook, not the model file — is the artifact your CI/CD system operates on.

4. Designing the DAG: small steps, durable handoffs, idempotent by default

Nearly every orchestrator — Airflow, Kubeflow Pipelines, SageMaker Pipelines, Azure ML, Dagster, Prefect — expresses a workflow as a directed acyclic graph: tasks as nodes, dependencies as edges, no cycles, so execution order is always derivable. The interesting design questions are not about the formalism; they are about granularity, data handoff, and re-execution.

Granularity. Cut the DAG along the natural stage boundaries — ingestion, validation, feature engineering, training, evaluation, registration, deployment — and resist monolithic nodes that fuse unrelated work. Azure's guidance captures the payoff: steps map to specific tasks so teams can work independently, and modular pipelines "reuse outputs from unchanged steps and let you run each step on the best compute resource for the task"^[3]. A fused preprocess-and-train node cannot skip the preprocessing on a retry, cannot cache, and cannot put feature engineering on cheap CPUs while training gets GPUs.

Handoffs. Pass data between steps through durable storage — object stores, feature stores, registries — never through in-memory coupling between tasks. Durable handoffs are what make a step independently re-runnable: when step six fails, you restart step six against its persisted inputs instead of replaying the whole graph. They are also what make backfills possible at all.

Idempotency. Design every step so that running it twice with the same inputs produces the same state — write outputs to deterministic, versioned paths and make final writes atomic. Idempotency is the precondition for everything in the error-handling section below: a retry policy attached to a non-idempotent step is a data-corruption feature. For long-running training steps where a full rerun is expensive, checkpointing is the complement — retries resume from the last checkpoint instead of from zero.

One more structural decision: which pipeline layer are you actually building? Microsoft's own comparison is clarifying — it distinguishes model orchestration (Azure ML pipelines; open-source analog Kubeflow Pipelines; canonical flow data to model), data orchestration (Azure Data Factory; analog Apache Airflow; data to data), and code-and-app CI/CD (Azure Pipelines; analog Jenkins; code plus model to app or service)^[3]. Teams that force all three concerns into one Airflow instance end up with a DAG repository nobody can safely change. Keep the layers separable, and let the ML pipeline consume the data pipeline's outputs through a contract, not a shared scheduler.

5. Event-driven ML: when the batch window closes

Batch, streaming, and real-time are not a maturity ladder; they are latency tiers, and each tier costs more to operate than the one before. A scheduled batch DAG is right when model freshness is measured in hours or days. You move to event-driven architecture when a business SLA — fraud scoring, dynamic pricing, live personalization — requires features or predictions that reflect what happened seconds ago. The decision input is the freshness SLA, not architectural fashion.

The event-driven pattern itself is consistent across stacks: an append-only event backbone decouples producers from consumers; stream processors derive features from the event flow; and pipeline stages communicate through topics rather than through an orchestrator's control plane. Google's Pub/Sub documentation states the decoupling premise plainly: it is "an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages," with publishers sending events "without regard to how or when these events are to be processed"^[6]. That indifference is the architectural point — the training trigger, the feature updater, and the monitoring consumer can all subscribe to the same stream without knowing about each other.

For the backbone, the enterprise-realistic choices are managed services. Amazon MSK is "a streaming data service that manages Apache Kafka infrastructure and operations," aimed at teams that want Kafka semantics without operating Kafka clusters^[8]. Azure Event Hubs takes a different route to the same destination: a cloud-native broker with "built-in Apache Kafka compatibility" that lets you "run existing Kafka workloads without code changes or cluster management overhead," with an SLA of up to 99.99% depending on tier^[5]. On Google Cloud, Pub/Sub pairs with Dataflow, "a Google Cloud service that provides unified stream and batch data processing at scale" built on Apache Beam, which the documentation positions explicitly for "real-time machine learning (ML) analysis of streaming data"^[9]. If your feature pipeline must exist in both batch and streaming forms, Beam's unified model is the strongest argument in the Google column — one pipeline definition, two execution modes, no logic drift between them.

Operationally, three practices keep event-driven ML pipelines honest. Enforce schemas at the topic boundary — Event Hubs, for instance, ships a schema registry to keep producers and consumers compatible as formats evolve^[5] — because in a streaming system a silent schema break corrupts features continuously, not once per batch. Track consumer lag as a first-class SLO, since lag is the streaming equivalent of a missed batch window. And keep the orchestrated and event-driven worlds connected deliberately: a common hybrid runs streaming feature computation continuously while an orchestrated DAG performs retraining, triggered when enough new data has accumulated. The same discipline applies downstream at serving time, where the cost and latency tradeoffs are covered in /guides/llm-inference-at-scale.

6. Error handling: a failure taxonomy, then a retry policy

Retry logic is where ML pipelines quietly burn money. The discipline that prevents it is ordering: classify failures first, then attach retry behavior per class. A transient fault — a throttled API, a brief network partition, a capacity shortage — is worth retrying. A permanent fault — a schema violation, a broken container, a bug — is not, and retrying it just adds cost and delays the page to a human. The vendor taxonomies make this concrete: SageMaker Pipelines defines distinct retryable exception classes such as Step.SERVICE_FAULT and Step.THROTTLING for transient downstream faults, and SageMaker.CAPACITY_ERROR and SageMaker.RESOURCE_LIMIT for compute shortages where waiting and retrying can genuinely succeed^[4]. Step Functions draws the same line from the other side: a States.Runtime error "isn't retriable, and will always cause the execution to fail"^[2]. Your internal pipeline code deserves the same explicitness — structured errors with codes, so the orchestrator can decide mechanically which class it is looking at.

The retry mechanics themselves are well standardized, and AWS Step Functions documents the canonical parameter set: an initial interval before the first retry (default 1 second), a maximum number of attempts (default 3), and a backoff rate multiplying the interval after each attempt (default 2.0), optionally capped by a maximum delay^[2]. Exponential backoff alone is not enough at fleet scale — when many failed tasks share a clock, their synchronized retries arrive as a thundering herd. That is what jitter is for; in AWS's words, "jitter reduces simultaneous retry attempts by spreading these out over a randomized delay interval"^[2]. Jitter is opt-in there (the default strategy is NONE), and turning it on for any high-fan-out step is close to free insurance.

Bound every retry policy twice: by attempts and by blast radius. SageMaker's step-level retry policy defaults to 5 attempts, hard-caps at 20, and alternatively supports an expiry window — retries stop after a set number of minutes regardless of attempt count^[4]. The second bound matters more than the first for expensive steps: five retries of a flaky metadata call cost nothing, while five retries of a multi-hour GPU training job is a five-figure incident. Scale retry aggressiveness inversely with step cost, and pair retries with checkpointing so an expensive step resumes rather than restarts.

Retries handle the failure; catch-and-fallback handles the aftermath. By default, "when a state reports an error, Step Functions defaults to failing the entire state machine execution"^[2] — and that fail-the-world default is what catch handlers exist to override. The production-grade patterns are graceful degradation (route to a recovery state, serve the last known-good model, skip a non-critical enrichment) and circuit breaking (stop retrying a dependency that is failing at a high rate before you amplify the outage). Every fallback should emit a loud, structured signal: a pipeline silently serving last week's model is an incident that has not been reported yet.

7. Honest objections

"Level 2 MLOps is overkill for us." Often true. If you run two models, retrain quarterly, and your data drifts slowly, a level 0-to-1 posture — a versioned, automated training pipeline without full CI/CD around the pipeline itself — may be the correct steady state. Google's maturity model is descriptive, not a mandate; the levels exist so you can name your position and its costs, and the guide itself frames level 2 around teams whose data scientists need to iterate rapidly on new ideas^[1]. If nobody is iterating rapidly, do not build the machinery for it.

"Event-driven is operational quicksand." The burden is real: schema evolution, consumer lag, event ordering, exactly-once questions, and a debugging model that is genuinely harder than reading a DAG run log. Managed backbones remove cluster operations, not distributed-systems reasoning. The honest test is whether a business SLA actually requires sub-minute freshness. If your fraud model can be an hour stale without measurable loss, a frequent batch schedule is simpler, cheaper, and easier to audit — and choosing it is engineering judgment, not conservatism.

"Aggressive retries paper over a bad platform." Partly right. Retries are a tax you pay for operating on shared, rate-limited, occasionally failing infrastructure — and every mature platform pays it, which is why the cloud vendors ship retry machinery as a first-class primitive^[4]. But a rising retry rate is a signal, not noise. If your dashboards show retry counts climbing month over month, the correct response is root-cause work on the flaky dependency, not raising the attempt cap. Track retries as a health metric with the same seriousness as failures.

8. The read

For a platform lead, the sequencing is: first make the training pipeline the deployable, versioned unit (level 1 — this is the highest-leverage move on the board); second, invest in DAG hygiene — small steps, durable handoffs, idempotency — because every later capability, from caching to retries to backfills, is built on it; third, add CI/CD around the pipeline when experiment throughput, team count, or audit pressure justifies it; and only then go event-driven, use-case by use-case, where a freshness SLA pays for the operational complexity. Error handling is not a phase — it is a property you hold constant across all four moves. And instrument as you go: the observability and release-management side of running models and LLM systems in production is its own discipline, covered in /guides/llm-observability-and-releases.

On the build-versus-buy question, the managed orchestrators (SageMaker Pipelines, Azure ML pipelines, and their peers) have absorbed the undifferentiated heavy lifting — serverless execution, lineage, retry primitives, step caching^[7]. Self-managing an orchestrator is defensible when you need multi-cloud portability or have platform engineering as a core competency; otherwise the differentiation is in your DAG design, your validation gates, and your failure taxonomy — all of which are portable across orchestrators, and none of which any vendor can supply for you.