Deductive reasoning involves using general principles or rules to draw specific conclusions, while inductive reasoning involves drawing general conclusions from specific instances or examples. Both types of reasoning are crucial for intelligence, but most research on LLMs does not make a clear distinction between their deductive and inductive capabilities.
To evaluate the reasoning abilities of LLMs, the researchers designed a series of experiments that focused on either deductive or inductive reasoning. For example, in an arithmetic task, they tested the LLMs’ ability to apply a given mathematical function (deductive reasoning) and their ability to infer the underlying function from examples (inductive reasoning).
To isolate and evaluate the inductive reasoning process in LLMs, the researchers developed a two-step framework called SolverLearner. In the first step, the LLM is prompted to generate a function based on input-output examples, focusing on its ability to learn patterns or rules from the data. In the second step, an external code interpreter executes the proposed function on new test data, ensuring that deductive reasoning does not influence the evaluation of inductive reasoning.
Using SolverLearner, the researchers evaluated the inductive and deductive reasoning capabilities of two LLMs, GPT-3.5 and GPT-4. The results showed that both models excelled in inductive reasoning, achieving near-perfect accuracy in tasks that required learning from examples and inferring underlying patterns. However, they struggled with applying specific rules or instructions, particularly in scenarios not encountered during their training.
Notably, the LLMs performed well in deductive reasoning tasks involving base 10 arithmetic but struggled with unconventional numerical bases. This suggests that LLMs are better at learning from examples and discovering patterns in data than following explicit instructions. As a result, their performance may degrade when faced with examples that deviate from their training distribution.
The findings have important implications for the use of LLMs in real-world scenarios. While they may appear to follow logical instructions, they may actually be relying on patterns observed during training. Therefore, their performance may suffer when encountering novel situations. On the other hand, SolverLearner provides a framework to ensure that LLMs learn the correct rules for mapping inputs to outputs, but it requires a verification mechanism like a code interpreter.
In conclusion, this study highlights the need for a deeper understanding of LLMs’ reasoning abilities. While they demonstrate impressive capabilities, there is still much to learn about these black box models that are being integrated into various applications. As we continue to explore their potential, it is crucial to consider their limitations and the importance of training them in diverse scenarios.
