Enkrypt, a startup specializing in the safe use of generative AI, has created a Large Language Model (LLM) Safety Leaderboard to help enterprises determine the safest and most reliable LLM for their applications. The leaderboard ranks LLMs based on their vulnerability to safety and reliability risks and provides valuable insights for choosing the best model.
Ensuring the safety of LLMs is crucial, as even a small error in testing and evaluation efforts can lead to broken user experiences, lost opportunities, and even regulatory fines. For example, Google’s Gemini chatbot suffered from biased outputs, highlighting the need for robust safety measures, particularly in regulated industries like fintech and healthcare.
Enkrypt has been addressing this problem with its comprehensive solution, Sentry, which identifies vulnerabilities in gen AI apps and deploys automated guardrails to prevent them. The LLM Safety Leaderboard is an extension of this work and offers a risk score for 36 open and closed-source LLMs. The score considers various safety and security metrics, including the model’s ability to avoid generating harmful or biased content and its effectiveness in blocking malware or prompt injection attacks.
As of May 8, Enkrypt’s leaderboard ranks OpenAI’s GPT-4-Turbo as the safest LLM with a risk score of 15.23. The model excels in defending against jailbreak attacks and provides toxic outputs only 0.86% of the time. However, it does face issues of bias and malware in a significant percentage of cases.
Other models that perform well on the leaderboard include Meta’s Llama2 and Llama 3 family, which have risk scores ranging from 23.09 to 35.69. Anthropic’s Claude 3 Haiku also ranks 10th with a risk score of 34.83. These models perform decently across most tests but struggle with bias in certain cases.
On the other end of the spectrum, models like Saul Instruct-V1 and Microsoft’s Phi3-Mini-4K have higher risk scores of 60.44 and 54.16, respectively. Mixtral 8X22B and Snowflake Arctic also rank low on the leaderboard with risk scores of 28 and 27.
It’s important to note that the leaderboard will evolve as existing models improve and new ones enter the market. Enkrypt plans to update the leaderboard regularly to reflect these changes and provide up-to-date information.
According to Sahi Agarwal, the co-founder of Enkrypt, integrating the leaderboard into AI strategy not only enhances technological capabilities but also upholds ethical standards and builds trust. Enterprise teams can use the leaderboard to understand the strengths and weaknesses of each LLM and make informed decisions based on their specific use case. This level of information goes beyond public performance benchmarks and provides valuable safety recommendations for deploying models.
In conclusion, Enkrypt’s LLM Safety Leaderboard offers a valuable resource for enterprises looking to integrate AI responsibly. By considering safety and reliability risks, the leaderboard helps companies choose the most secure LLM for their applications, ensuring the protection of user data and maintaining ethical standards.