While major companies trend toward massive models spanning hundreds of billions of parameters, a new current focused on “small language models” with fewer parameters has begun to emerge. Why this shift, and what are the advantages?
Article contents:
- From Hebb to Rosenblatt: psychological roots
- Drawbacks and costs of massive models
- What distinguishes small models
- Core technology: distillation and pruning
- Future prospects
- Frequently asked questions
From Hebb to Rosenblatt: psychological roots
The beginnings of artificial neural networks date back to psychological research in the 1950s, such as the work of Donald Hebb, who proposed the principle of strengthening neural connections when cell activity coincides. Frank Rosenblatt then developed the “perceptron” model, inspired by brain learning mechanisms, paving the way for deep networks.
Drawbacks and costs of massive models
Despite their power, massive models require supercomputing and heavy energy consumption. Reports have indicated that Google spent approximately $191 million to train the Gemini 1.0 Ultra model. These models also consume immense energy with every response, making them unsuitable for everyone.
What distinguishes small models
They use fewer parameters, which reduces training cost and energy. They excel in specialized tasks if trained on high-quality data. As Zico Kolter from Carnegie Mellon University points out, “For many tasks, an 8-billion-parameter model is enough.”
Core technology: distillation and pruning
The small model learns from the larger model through what is called “knowledge distillation,” where high-quality data is provided. “Pruning” can also be applied to remove unnecessary units from the massive model without a significant loss in performance.
Future prospects
Researchers expect the spread of small models to provide lightweight and fast solutions, particularly in smartphones and resource-constrained devices, and among startups that cannot afford the cost of giant models. Massive models remain important for large-scale tasks, but small ones constitute a practical alternative for specific tasks.
Frequently asked questions
1. Are small models always less accurate?
Not necessarily; they may outperform in specific tasks thanks to training focused on pure and highly relevant data for the subject of the task.
2. How can a startup benefit?
It can adopt an open-source or low-cost small model and train it on its own data, which lowers the infrastructure bill.
3. Will this replace giant models?
No, as massive models remain essential for general and complex tasks, while small ones make an ideal choice for specialized tasks.
4. What is the exact role of knowledge distillation?
It is a process in which a small model extracts useful information from a large model, saving it time and resources.
5. Are there prominent examples of small model success?
Yes, several companies have released models with relatively few parameters that have achieved remarkable results in analyzing specific texts or summarizing conversations.
