Large language models (LLMs) have revolutionized various tasks, but they are not without their limitations. One major challenge with LLMs is their tendency to make factual and logical errors, especially in complex reasoning tasks. To address this issue, researchers have been using verifiers or reward models to evaluate and select the most accurate responses from a set of LLM-generated outputs. However, traditional verifiers and reward models have their own limitations and fail to fully leverage the generative capabilities of LLMs.
In a recent paper, researchers from Google DeepMind, the University of Toronto, Mila, and the University of California, Los Angeles introduced a novel approach called GenRM that aims to overcome the limitations of traditional verifiers and reward models. GenRM leverages the generative capabilities of LLMs to create more effective verifiers, making it a practical tool for LLM applications where current verification methods fall short.
Traditional verifiers rely on discriminative reward models (RMs) to assign numerical scores to candidate solutions and classify them as correct or incorrect. However, these RMs do not fully utilize the text generation capabilities of LLMs. On the other hand, LLM-as-a-Judge, another popular technique, uses advanced prompting techniques to evaluate responses but lacks the training abilities of reward models.
GenRM takes a different approach by training verifiers using next-token prediction to tap into the generative capabilities of LLMs. By representing the verification decision as a token, the verifier can produce a numerical score for a solution based on a prompt. This approach allows for more advanced prompting techniques like chain-of-thought (CoT) reasoning, where the model generates a thought process before providing an answer. By generating intermediate reasoning steps or critique before making a decision, GenRM can identify subtle reasoning errors that direct verifiers might miss.
To evaluate the effectiveness of GenRM, the researchers tested it on several reasoning tasks, including last-letter concatenation, word sorting, and word-math problems. They compared GenRM against standard approaches such as discriminative reward models, LLM-as-a-Judge, and self-consistency. The results showed that GenRM with CoT consistently outperformed the other methods by several percentage points.
For example, on the GSM8K math reasoning benchmark, a GenRM model solved 92.8% of the problems, surpassing the performance of GPT-4 and Gemini 1.5 Pro. The researchers also found that GenRM scales well with increasing dataset size and model capacity. Additionally, allowing GenRM to sample more responses further improves its performance, giving developers more flexibility to balance accuracy and compute costs.
While GenRM shows promising results, there are still future directions to explore. These include scaling synthetic verification rationales on open-ended generation tasks, integrating GenRMs into reinforcement learning pipelines, and leveraging other advanced LLM capabilities to enhance verification.
In conclusion, GenRM is a novel approach that leverages the generative capabilities of LLMs to create more effective verifiers. It overcomes the limitations of traditional verifiers and reward models by enabling verifiers to tap into the benefits of LLMs’ text generation capabilities. The experiments conducted by the researchers demonstrate that GenRM outperforms other verification methods, making it a valuable tool for various LLM applications.
