Right-Sizing Your Models: Fine-Tuning, Distillation, Pruning, and Small Language Models

Right-sizing a model portfolio means matching capability to task, not defaulting to the largest frontier model for everything. The ladder runs: prompting and retrieval first, managed fine-tuning when evaluations prove a gap, distillation when volume justifies training a student, quantization and pruning when self-hosted serving costs bite, and a small language model when a 1-8B open-weight model genuinely clears your quality bar.

This guide is the decision framework for that ladder. Each rung trades capability headroom for cost, latency, or control — and each rung adds an ownership burden that is easy to underestimate. The recurring theme: the constraint is rarely the training bill. It is the evaluation infrastructure you need to prove, continuously, that the smaller or cheaper thing is still good enough.

1. By the numbers

2. The right-sizing ladder

The mistake most teams make is treating these techniques as competing options. They are sequential rungs, and each one is only worth climbing when the rung below has demonstrably failed your evaluation criteria. Skipping ahead — fine-tuning before prompting has plateaued, or self-hosting a compressed model before the API bill justifies the platform team — buys ownership costs without evidence they were needed.

3. Rung zero: prompting and retrieval are the control group

The vendors selling fine-tuning are unusually candid about this. OpenAI's model-optimization guide positions evals as the foundation — measure first, then iterate — and states plainly that "the prompt engineering process may be all you need in order to get great results for your use case."^[5] That is the correct default posture for an enterprise: prompting and retrieval-augmented generation change the input, not the model, so they carry no training spend, no snapshot pinning, and no retraining cycle when the base model improves underneath you.

The strategic reason to exhaust this rung first is not just cost. A prompt-plus-retrieval architecture is portable across models and vendors; a fine-tune is not. Every dollar you invest in better retrieval, better context assembly, and better eval sets transfers when you switch models. Every dollar in a fine-tune is married to one vendor's one dated snapshot. Before any customization spend, the honest question is: have we actually plateaued on prompting, and can we show the plateau in an eval, not an anecdote?

4. The vendor reality check: fine-tuning availability is itself a moving target

Before modeling fine-tuning costs, check whether your vendor still sells it. The three major model providers have diverged sharply, and the divergence is a governance fact your architecture should absorb.

Read that table as a risk signal, not just a feature matrix. Google shipped API fine-tuning, then removed it when the one tunable model was deprecated.^[6] Anthropic offers tuning only through a hyperscaler channel, for one small model.^[7] Only OpenAI treats managed fine-tuning as a mainline product today — and even there, tunes attach to dated snapshots like gpt-4.1-2025-04-14, which will eventually age out on the vendor's schedule, not yours.^[1] If your differentiation strategy depends on customized weights, the availability of customization is itself a vendor-concentration exposure. That argues for keeping your training data and eval sets vendor-neutral, so the asset survives even when a specific tuning product does not.

5. What fine-tuning actually costs: data prep dominates, training does not

The published training prices are the most visible and least important part of the bill. OpenAI lists supervised fine-tuning at $25.00 per 1M training tokens for gpt-4.1, $5.00 for gpt-4.1-mini, and $1.50 for gpt-4.1-nano; reinforcement fine-tuning on o4-mini is priced differently, at $100.00 per hour of training.^[1] Against enterprise budgets these are small numbers — a modest tuning dataset re-run several times for hyperparameter iteration still lands in the hundreds or low thousands of dollars of compute.

The real spend sits on either side of that bar chart. Upstream is data preparation: collecting, cleaning, deduplicating, and labeling domain examples, plus building the held-out eval set that tells you whether the tune worked. This is skilled human labor — subject-matter experts writing and reviewing demonstrations — and it recurs every time the task definition drifts or the base snapshot is deprecated. Notably, the bottleneck is quality rather than volume: OpenAI's guide reports that improvements typically appear with 50-100 examples and recommends starting from 50 well-crafted demonstrations.^[2] Fifty excellent examples authored and adjudicated by your best domain experts is a real project; a million scraped mediocre ones is a liability.

Downstream is the inference premium. Fine-tuned gpt-4.1-mini serves at $0.80 input and $3.20 output per 1M tokens against $0.40 and $1.60 for the base model — exactly double the listed base rates — and fine-tuned gpt-4.1-nano at $0.20/$0.80 against $0.10/$0.40 base.^[1] So a fine-tune must earn back a 2x per-token markup through accuracy, shorter prompts (tuned behavior replaces lengthy instructions and few-shot examples), or a step down in model size that still clears your eval bar. That last pattern — tune the smaller model until it matches the bigger one on your narrow task — is where the economics usually work, and it leads directly to the next rung.

6. Distillation: the forty-year-old idea the vendors productized

Distillation — training a small student model to reproduce a large teacher's behavior — predates the LLM era. The canonical paper is Hinton, Vinyals, and Dean's 2015 "Distilling the Knowledge in a Neural Network," which built on the observation, credited to Caruana and collaborators, "that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy," and demonstrated it by improving "the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model."^[8]

The modern managed version is prosaic: use the big model to write the small model's training data. OpenAI's fine-tuning guide describes the loop explicitly — tune a prompt for a larger model such as gpt-4.1 until it performs well against your eval criteria, capture its production responses (the Responses API stores them for 30 days by default), filter to the outputs that meet your bar, then fine-tune a smaller model such as gpt-4.1-mini on that dataset. The stated payoff: "this technique can enable you to train a small model to perform similarly on a specific task to a larger, more costly model."^[2]

When does that loop clear its own costs? Three conditions, all of which should hold at once. First, volume: the teacher-to-student price gap is only worth harvesting when the task runs at sustained scale, because you pay the teacher's rates to generate training data and pay engineers to run the loop. Second, stability: the task definition must sit still long enough to amortize the effort — a distilled student encodes yesterday's task, and every task change means re-generating data and re-tuning. Third, narrowness: distillation transfers performance on a specific distribution, not general capability, so the student will be brittle exactly where your traffic wanders off the training distribution. High-volume, stable, narrow — a classification pipeline, a fixed-schema extraction task, a templated drafting flow — is the profile. An open-ended assistant is the anti-profile.

7. Quantization and pruning: compressing what you already host

The compression rungs only apply once you self-host open weights — for API consumers, the vendor has already made these choices. For self-hosters, quantization is the mature, near-default technique: reducing weight precision from 16-bit floats to 8 or 4 bits to cut memory and cost. The two load-bearing research results are worth knowing by name. LLM.int8() showed that 8-bit inference can "cut the memory needed for inference by half while retaining full precision performance" for transformers up to 175B parameters, by isolating outlier features into a small 16-bit path while keeping more than 99.9% of values in 8-bit.^[9] GPTQ pushed further, quantizing GPT-class models "down to 3 or 4 bits per weight, with negligible accuracy degradation" in about four GPU hours for a 175B model, with reported end-to-end speedups of around 3.25x on an NVIDIA A100 and 4.5x on an A6000.^[3]

The practical translation for a platform lead: 4-bit quantization roughly quarters the GPU memory a model needs versus 16-bit serving, which means fitting a given model on fewer or cheaper GPUs, or a bigger model on the GPUs you have. It is the highest-leverage, lowest-drama optimization in self-hosted serving — but "negligible degradation" is a claim about the papers' benchmarks, not about your workload. Quantized variants must go back through your task evals, and calibration-sensitive applications should re-check confidence behavior, not just accuracy.

Pruning — removing weights outright — is the less deployable sibling. Unstructured pruning zeroes individual weights and can reach high sparsity, but realizing actual speedups requires sparse-kernel support that mainstream GPU serving stacks mostly lack; structured pruning removes whole neurons or attention heads and runs fine on standard hardware, but yields more modest compression before quality drops. For most enterprise teams the honest sequencing is: quantize first, prune only if you have the specialized serving expertise to exploit it, and treat both as serving-stack decisions rather than model-strategy decisions. The broader serving economics — batching, caching, GPU utilization — are covered in the companion piece at /guides/llm-inference-at-scale.

8. The SLM case: when 1-14B parameters is genuinely enough

The final rung skips compression tricks entirely and changes the model class. Small open-weight models have become good enough, and permissively licensed enough, that for closed-domain tasks they are a legitimate endpoint rather than a compromise. The claims below come from the models' own cards, which is the right sourcing discipline in a segment where names and versions churn fast.

The pattern in the licensing column matters as much as the parameter counts. MIT-licensed Phi-4 models carry essentially no usage encumbrance; Llama 3.2 ships under Meta's community license, which is workable for most enterprises but is a bespoke agreement your counsel reads, not a standard OSI license.^[10][4] The full open-weight governance calculus — license review, provenance, indemnification, support — is treated in depth at /guides/open-source-ai-enterprise-guide.

Where does a 1-8B model genuinely suffice? The reliable envelope is closed-domain work: classification and routing, entity and schema extraction, template-bound generation, summarization of bounded documents, on-device or in-VPC assistants where data residency forbids external APIs, and high-volume pipeline steps where a frontier model would be economically absurd. The envelope's edge is open-ended reasoning: multi-step planning, novel synthesis across domains, and agentic workflows that recover from their own errors remain the territory of large models — and of reasoning models specifically, whose distinct economics are covered at /guides/reasoning-models-enterprise-guide. The right mental model is a portfolio: small models as the workhorses on high-volume narrow lanes, frontier models reserved for the tasks that measurably need them, with routing between the two as an explicit architectural layer.

9. Honest objections: the ladder's hidden invoices

Objection one: customization is a maintenance treadmill, not an asset

The strongest case against climbing the ladder is temporal. A fine-tune binds to a dated snapshot — gpt-4.1-2025-04-14, o4-mini-2025-04-16^[1] — and base models improve on a cadence you do not control. Every base-model release forces a question: does the new base model with plain prompting now beat your tuned old one? Often it does, which means the tuning investment must be re-made or written off — and the same re-tuning cycle applies to distilled students and quantized variants whenever the upstream model or the task moves. The Gemini API is the cautionary tale in production form: teams that tuned Gemini 1.5 Flash-001 saw the only tunable model deprecated in May 2025 and API fine-tuning withdrawn with it.^[6] The counter-position is not "never customize" — it is that the durable asset is your curated dataset and eval suite, which survive every snapshot deprecation, while the tuned weights themselves are a depreciating artifact. Budget customization as an operating expense with a short amortization window, never as a one-time capital project.

Objection two: proving "good enough" costs more than you think

Every rung of the ladder rests on the claim that the smaller or cheaper artifact is "good enough," and that claim is expensive to establish. A credible eval program needs representative task data, human-adjudicated ground truth, regression coverage for the failure modes that matter (including safety and formatting, not just accuracy), and re-runs on every model change — yours or the vendor's. For a low-volume workload, that evaluation program can cost more than the inference savings it is meant to unlock. The steelman conclusion deserves stating plainly: below a real volume threshold, the cheapest total system is the biggest capable model on a pay-as-you-go API with good prompts, because it converts all of this fixed evaluation and engineering cost into a variable per-token cost. Right-sizing is a scale play. Compute your own threshold with the unit-economics framework at /guides/llm-finops-guide before assuming the ladder pays.

10. The read: capability is a budget line, so provision it like one

The decision this guide supports is a portfolio posture, not a technique choice. Default every new use case to rung zero — a capable API model, strong prompting, retrieval — and require eval evidence, not enthusiasm, to fund each climb. Treat managed fine-tuning as a targeted instrument for formatting, tone, and narrow accuracy on a proven-stable task, priced with its 2x inference premium and snapshot-deprecation risk in view.^[1] Reserve distillation for the high-volume, stable, narrow lanes where the teacher's bill is demonstrably the dominant cost. Apply quantization the moment you self-host — it is close to free performance^[3][9] — and treat pruning as a specialist's tool. Adopt SLMs where the task envelope is closed and the card-level facts (license, context, size) fit your constraints.^[10][4] And hold one asset above all of it: the vendor-neutral eval suite and curated task dataset, because every rung of the ladder depreciates except that one.