Model distillation is a training technique where a small 'student' model learns to copy the outputs of a large 'teacher' model. The result is a faster, cheaper model that keeps most of the original's accuracy for a specific task.
Why it exists
Big models are expensive to run: more compute, more latency, more cost per query. Distillation lets a company keep the intelligence of a large model while shrinking the bill for actually serving it, which matters at scale where every millisecond and every GPU-hour adds up.
How it works
You run the large teacher model on a set of inputs and record its outputs, including its confidence patterns, not just the final answer. A smaller student model is then trained to match those outputs, effectively learning the teacher's behavior instead of learning from raw data alone.
Where it matters in commerce
Distilled models are common in production systems like recommendation engines, search ranking, and customer-facing chatbots, where you need near-instant responses at high volume. It's also how many companies deploy fine-tuned capabilities on-device or at lower cost without building a giant model from scratch.
The tradeoff
Distillation is not free knowledge transfer; the student typically performs slightly worse than the teacher, especially on edge cases outside the training examples. Leaders choosing this route are betting that the cost and speed gains outweigh a small accuracy loss for their specific use case.
Frequently asked
Is distillation the same as fine-tuning?
No. Fine-tuning adjusts a model using labeled data for a task, while distillation trains a smaller model to imitate a larger model's outputs.
Does distillation always shrink model size?
Usually yes, that's the main point, but the technique can also be used to transfer specific skills without necessarily changing size.
Why would a company use a distilled model instead of the original?
Lower inference cost, faster responses, and easier deployment on limited hardware, often with only a small accuracy tradeoff.