OpenAI is taking steps to enhance the safety and alignment of AI models with the introduction of Rules-Based Rewards (RBR). This new approach automates certain aspects of model fine-tuning, reducing the time required to ensure that models do not produce unintended results. According to Lilian Weng, head of safety systems at OpenAI, while reinforcement learning from human feedback has been the traditional method of training models, it often involves lengthy discussions about policy nuances, which can result in policies evolving before they are finalized.
RBR offers a more efficient way to align AI models with desired safety policies. Instead of relying solely on reinforcement learning from human feedback, OpenAI’s safety and policy teams use an AI model to score responses based on their adherence to a set of predefined rules. For example, a mental health app’s model development team may want the AI model to reject unsafe prompts in a non-judgmental manner, while also encouraging users to seek help when needed. To achieve this, they would create three rules for the model: rejecting the request, sounding non-judgmental, and using encouraging language.
The RBR model evaluates responses from the mental health model and checks if they align with the three rules. OpenAI claims that testing models using RBR produces results comparable to human-led reinforcement learning. However, there are challenges in ensuring that AI models respond within specific parameters. Failures can lead to controversies, as seen with Google’s Gemini model overcorrecting image generation restrictions.
One of the advantages of RBR is that it reduces subjectivity compared to human evaluators. Weng argues that ambiguous instructions lead to lower-quality data, as trainers struggle to interpret the meaning of “safe.” By narrowing down instructions and providing clear rules, RBR can improve the quality of data used for training models.
OpenAI acknowledges that RBR may reduce human oversight, which raises ethical considerations and the potential for increased bias in models. The company suggests combining RBRs with human feedback to ensure fairness and accuracy. However, RBR may face challenges with subjective tasks, such as writing and creative endeavors.
OpenAI began exploring RBR methods during the development of GPT-4, and Weng states that RBR has significantly evolved since then. The initiative aims to address concerns about OpenAI’s commitment to safety, as raised by former researcher Jan Leike, who criticized the company’s safety culture and processes. Co-founder and chief scientist Ilya Sutskever, who co-led the Superalignment team, also left OpenAI to start a new company focused on safe AI systems.
Overall, OpenAI’s introduction of Rules-Based Rewards represents a significant step towards enhancing the safety and alignment of AI models. While challenges remain, RBR offers a more streamlined approach that can reduce subjectivity and improve the quality of data used for training. The integration of RBRs with human feedback can help ensure fairness and accuracy in AI systems.