Large language models (LLMs) are incredibly proficient at answering simple questions but struggle with complex tasks that require reasoning and planning. To address this limitation, researchers at Meta FAIR have developed a technique called “System 2 distillation.” By distilling the knowledge gained from LLMs’ own System 2 reasoning capabilities into their fast-paced System 1 generation, they aim to improve their performance on complex tasks without sacrificing speed or computational efficiency.
Understanding System 1 and System 2 Thinking
In cognitive science, System 1 and System 2 refer to two distinct modes of thinking. System 1 is fast, intuitive, and automatic, while System 2 is slow, deliberate, and analytical. LLMs are typically associated with System 1 thinking, as they can generate text quickly but struggle with tasks that require deliberate reasoning and planning.
The Role of System 2 Prompting Techniques
Recent advancements in AI research have shown that LLMs can be prompted to mimic System 2 thinking by generating intermediate reasoning steps before providing their final answer. Techniques like “Chain of Thought” instruct LLMs to explain their reasoning process step by step, leading to more accurate results for logical reasoning tasks. However, these techniques come at the cost of increased inference time and latency, making them unsuitable for production systems that prioritize speed (System 1 generation).
Introducing System 2 Distillation
The inspiration behind System 2 distillation comes from the observation that humans can gradually internalize complex tasks that initially require deliberate effort. Similarly, the Meta AI researchers aimed to distill the knowledge gained from LLMs’ System 2 reasoning into their System 1 generation. Instead of using a separate teacher model for distillation, the researchers devised a method to verify the correctness of LLM responses through unsupervised mechanisms.
The Distillation Process
The process begins by prompting LLMs to solve problems using System 2 prompting techniques. The responses are then evaluated for consistency through self-comparison, selecting the most frequent answer as the correct one. Inconsistent answers are discarded. The intermediate reasoning steps generated by System 2 are then discarded, and only the final answers are kept. Finally, the model is fine-tuned on the initial question and answer, enabling it to skip reasoning steps and directly provide the answer.
Evaluating System 2 Distillation
The researchers evaluated their method using various reasoning tasks and four different System 2 prompting techniques. The results demonstrated that System 2 distillation significantly improved LLMs’ performance on complex tasks, often matching or surpassing the accuracy of the original System 2 methods. Furthermore, distilled models were able to generate responses much faster and with less compute, as they no longer needed to go through intermediate reasoning steps.
Limitations and Future Research
Although System 2 distillation showed promising results, the researchers discovered that not all types of reasoning skills could be successfully distilled into LLMs’ fast-paced inference mechanism. Some tasks, such as complex math reasoning requiring Chain-of-Thought prompting, may always require deliberate reasoning. Further research is needed to explore how well distillation works on smaller models and its impact on broader performance. Additionally, care must be taken to prevent contamination in LLM benchmarks, where the model already has prior knowledge of test examples.
Optimizing LLM Pipelines with Distillation
System 2 distillation has the potential to be a powerful optimization tool for mature LLM pipelines that perform specific tasks at each step. By distilling useful tasks into their fast-paced generation, LLMs can free up more time for reasoning about tasks they are not yet proficient in, similar to how humans approach complex problem-solving.
In conclusion, System 2 distillation offers a promising approach to enhancing the reasoning capabilities of LLMs. By incorporating their own System 2 reasoning into the fast-paced System 1 generation, LLMs can tackle complex tasks more effectively while maintaining speed and computational efficiency. Further research and refinement of this technique will undoubtedly contribute to the continued advancement of AI and its applications.
