The race among tech companies to develop on-device AI is leading to significant advancements in creating small language models (SLMs) that can run efficiently on resource-constrained devices. Nvidia’s research team has recently developed Llama-3.1-Minitron 4B, a compressed version of the Llama 3 model, using pruning and distillation techniques.
Pruning involves removing less important components of a model, such as complete layers or specific elements like neurons and attention heads. Distillation, on the other hand, transfers knowledge from a large model (teacher) to a smaller, simplified model (student). Nvidia researchers combined pruning with classical knowledge distillation in a previous study, resulting in a 16% improvement in performance compared to training a smaller model from scratch.
Building on their previous work, the Nvidia team applied the same techniques to the Llama 3.1 8B model. They fine-tuned the unpruned model on a large dataset to correct for distribution shifts, which resulted in better guidance during distillation. Two types of pruning were then applied: depth-only pruning, removing 50% of the layers, and width-only pruning, removing 50% of the neurons from some dense layers. This led to the creation of two versions of the Llama-3.1-Minitron 4B model.
The pruned models were further fine-tuned using NeMo-Aligner, a toolkit that supports various alignment algorithms. The researchers evaluated the performance of the Llama-3.1-Minitron 4B models in instruction following, roleplay, retrieval-augmented generation, and function-calling tasks. Despite being trained on a smaller corpus, Llama-3.1-Minitron 4B performed comparably to other SLMs, such as Phi-2 2.7B and Gemma2 2.6B, which were trained on larger datasets.
The width-pruned version of the model has been released on Hugging Face under the Nvidia Open Model License, making it accessible for commercial use. This development highlights the cost-effectiveness of pruning and classical knowledge distillation in obtaining smaller but accurate LLMs. It also emphasizes the importance of the open-source community in advancing AI research.
In addition to Nvidia’s work, other notable contributions in the field include Sakana AI’s evolutionary model-merging algorithm, which allows for combining the strengths of different models without requiring extensive training resources. These advancements in optimizing and customizing LLMs at a fraction of the usual cost are significant for the industry and pave the way for more efficient AI applications on resource-constrained devices.