Large language models (LLMs) have revolutionized the field of artificial intelligence, excelling at tasks like generating text, translating languages, and creating creative content. However, these models remain opaque and difficult to understand, even for the researchers who train them. This lack of interpretability poses challenges when using LLMs in critical applications that require transparency and have a low tolerance for mistakes. To address this challenge, Google DeepMind has released Gemma Scope, a set of tools that provide insights into the decision-making process of Gemma 2 models.
Understanding LLM activations with sparse autoencoders
When an LLM receives an input, it processes the information through a network of artificial neurons. The values emitted by these neurons, known as “activations,” represent the model’s understanding of the input and guide its response. By studying these activations, researchers can gain insights into how LLMs process information and make decisions. However, interpreting these activations is a major challenge due to the billions of neurons in LLMs and the massive jumble of activation values produced at each layer of the model.
Using sparse autoencoders for interpretability
One of the leading methods for interpreting LLM activations is using sparse autoencoders (SAEs). SAEs are models that help interpret LLMs by studying the activations in their different layers. They are trained on the activations of a layer in a deep learning model, representing the input activations with a smaller set of features and then reconstructing the original activations from these features. This process allows SAEs to compress the dense activations into a more interpretable form, making it easier to understand which features activate different parts of the LLM.
Introducing Gemma Scope
DeepMind’s Gemma Scope takes a comprehensive approach to interpretability by providing SAEs for every layer and sublayer of its Gemma 2 2B and 9B models. It comprises more than 400 SAEs, representing over 30 million learned features. This extensive collection of SAEs allows researchers to study how different features evolve and interact across different layers of the LLM, providing a richer understanding of the model’s decision-making process.
The power of JumpReLU SAE
Gemma Scope utilizes DeepMind’s new architecture called JumpReLU SAE. Unlike previous SAE architectures that used the rectified linear unit (ReLU) function to enforce sparsity, JumpReLU allows the SAE to learn a different activation threshold for each feature. This change makes it easier for the SAE to strike a balance between detecting which features are present and estimating their strength. JumpReLU also helps keep sparsity low while increasing the reconstruction fidelity, addressing one of the challenges of SAEs.
Advancing robust and transparent LLMs
DeepMind has made Gemma Scope publicly available on Hugging Face, enabling researchers to use these tools in their work. The release of Gemma Scope aims to facilitate more ambitious interpretability research, which can help build more robust systems and develop better safeguards against potential risks associated with LLMs. SAEs, like those provided in Gemma Scope, offer promising avenues of research to block unwanted behavior in LLMs and detect issues such as generating harmful or biased content.
The broader landscape
Other organizations, such as Anthropic and OpenAI, are also working on their own SAE research and have released multiple papers in recent months. Additionally, scientists are exploring non-mechanistic techniques to understand LLMs’ inner workings. For example, OpenAI has developed a technique that pairs two models to verify each other’s responses, encouraging the model to provide verifiable and legible answers.
As LLMs continue to advance and find applications in various industries, the need for tools and research to understand and control their behavior becomes crucial. Gemma Scope and other interpretability efforts contribute to this goal, offering insights and techniques to ensure the robustness and transparency of LLMs.
