Pruning

Neural networks are significantly over-parameterized by design — they contain many more weights than are theoretically necessary to represent the learned function, because excess capacity during training helps avoid local minima and improves generalization. After training, many of these weights are close to zero or redundant with other weights, representing "dead" capacity that consumes memory and compute at inference time without contributing meaningfully to output quality. Pruning identifies and removes this redundancy.

Pruning methods divide into two families. **Unstructured pruning** removes individual weights wherever they fall below a magnitude threshold, creating a sparse weight matrix. While it can achieve high sparsity ratios (80–90% of weights removed), unstructured sparsity requires specialized sparse matrix hardware to realize speedups — conventional GPUs do not efficiently skip zero-valued multiplications without hardware support. **Structured pruning** removes entire neurons, attention heads, or layers — coarser-grained removals that produce a smaller dense network directly executable by standard hardware without special runtimes. For most enterprise inference deployments, structured pruning delivers more predictable, hardware-agnostic speedups.

Modern large language models are amenable to attention head pruning: empirical studies consistently show that a significant fraction of attention heads in transformer models are redundant for most tasks and can be removed with minimal accuracy impact. Enterprise teams pruning task-specific models (after fine-tuning on domain data) find that structured pruning of 20–40% of attention heads and feed-forward neurons can recover the same model in 60–70% of the original parameter count — meaningfully reducing memory pressure on inference servers and increasing the number of model instances that can be co-located on a given GPU node.