Quantization

Neural network models store their learned parameters as floating-point numbers. A typical 7-billion-parameter model in 16-bit (FP16) precision requires approximately 14 GB of GPU memory — already at the limit of many inference GPUs. Quantization replaces these high-precision floats with lower-precision integers. Moving from FP16 to INT8 halves memory requirements; moving to INT4 halves it again, enabling a 7B model to run in roughly 3.5 GB of GPU memory — or even on high-end consumer hardware.

There are two primary quantization strategies. **Post-training quantization (PTQ)** applies quantization to a pretrained model without retraining, making it fast and accessible — tools like GPTQ and AWQ implement sophisticated PTQ algorithms that calibrate quantization scales using a small representative dataset to minimize accuracy loss. **Quantization-aware training (QAT)** incorporates quantization into the training loop itself, simulating integer arithmetic during forward passes so the model learns to compensate — this achieves higher accuracy at a given precision level but requires access to the full training pipeline.

The practical enterprise decision tree is: start with PTQ (GPTQ or AWQ at INT4 or INT8) using your representative task data as calibration, measure accuracy on your internal benchmark, and proceed to QAT only if PTQ degradation is unacceptable. For most enterprise NLP tasks — classification, extraction, summarization, Q&A — INT8 PTQ achieves accuracy within 1–2% of the FP16 baseline while halving inference cost. INT4 methods (GPTQ, AWQ, GGUF) enable on-premise deployment on non-data-center hardware, opening edge and air-gapped deployment scenarios that were previously impractical.