Introduction:
In a recent study conducted by researchers at Meta, Ecole des Ponts ParisTech, and Université Paris-Saclay, a new technique for improving the accuracy and speed of AI large language models (LLMs) has been proposed. This technique involves making the models predict multiple tokens simultaneously instead of the traditional method of predicting one token at a time. While this approach may not be suitable for all types of models and language tasks, it has shown significant benefits in certain areas, such as triple speeds and better performance on generative tasks.
Limits of Next-Token Prediction:
The classic way to train LLMs is through next-token prediction, where the model predicts the next token in a sequence. However, this method has its limitations. Researchers have found that next-token prediction models struggle to acquire language, world knowledge, and reasoning capabilities. They often become too sensitive to local patterns and overlook predictions that require reasoning over longer horizons. Additionally, these models require large amounts of data to achieve fluency levels comparable to humans.
Multi-Token Prediction:
To address the limitations of next-token prediction, the researchers propose a multi-token prediction approach. This technique instructs the LLM to predict several future tokens from each position simultaneously, resulting in higher sample efficiency. The multi-token prediction language model is based on the Transformer architecture commonly used in LLMs but with some modifications. It utilizes multiple independent output heads to predict each token it wants to predict.
Transformer Architecture with Multi-Token Prediction:
During inference, the model uses the next-token prediction scheme for each prediction head, while leveraging the additional output heads to speed up the decoding process. This modification allows for stronger and faster transformer models without requiring extra training time or memory overhead. The researchers found that as the model size increases, multi-token prediction becomes increasingly useful, resulting in improved performance on various tasks.
Benefits of Multi-Token Prediction:
The study conducted by Meta and its collaborators revealed several benefits of multi-token prediction. Firstly, it makes models up to three times faster at inference time across a wide range of batch sizes. This speed improvement is crucial for applications that require real-time processing. Additionally, multi-token prediction promotes the learning of longer-term patterns, which is particularly valuable in scenarios where the model must work with small chunks of information or has no predefined vocabulary.
Future Directions and Enterprise Applications:
While multi-token prediction still has room for improvement, it holds promise for enterprise applications. It has the potential to provide faster inference and higher accuracy at little or no extra cost, especially for generative tasks like code completion. Furthermore, since multi-token prediction is compatible with other optimization techniques for the Transformer block, it can be seamlessly integrated into existing AI systems.
Conclusion:
The integration of AI in production is a complex process that requires responsible decision-making. The new technique of multi-token prediction offers a viable solution for improving the accuracy and speed of large language models. By predicting multiple tokens simultaneously, these models can overcome the limitations of next-token prediction and achieve better performance on various tasks. As researchers continue to explore and refine this approach, it is expected to have a significant impact on the future of AI in business.
