可解释性之梦

An informal note on future goals for mechanistic interpretability by Chris Olah. Published May 24th, 2023.

Our present research aims to create a foundation for mechanistic interpretability research. In particular, we're focused on trying to resolve the challenge of superposition. In doing so, it's important to keep sight of what we're trying to lay the foundations for. This essay summarizes those motivating aspirations – the exciting directions we hope will be possible if we can overcome the present challenges.

We aim to offer insight into our vision for addressing mechanistic interpretability's other challenges, especially scalability. Because we have focused on foundational issues, our longer-term path to scaling interpretability and tackling other challenges has often been obscure. By articulating this vision, we hope to clarify how we might resolve limitations, like analyzing massive neural networks, that might naively seem intractable in a mechanistic approach.

Before diving in, it's worth making a few small remarks. Firstly, essentially all the ideas in this essay were previously articulated, but buried in previous papers. Our goal is just to surface those implicit visions, largely by quoting relevant parts. Secondly, it's important to note that everything in this essay is almost definitionally extremely speculative and uncertain. It's far from clear that any of it will ultimately be possible. Finally, since the goal of this essay is to lay out our personal vision of what's inspiring to us,  it may come across as a bit grandiose – we hope that it can be understood as simply trying to communicate subjective excitement in an open way.

Overview

• An Epistemic Foundation – Mechanistic interpretability is a "microscopic" theory because it's trying to build a solid foundation for understanding higher-level structure, in an area where it's very easy for us as researchers to misunderstand.
• What Might We Build on Such a Foundation? – Many tantalizing possibilities for research exist (and have been preliminarily demonstrated in InceptionV1), if only we can resolve superposition and identify the right features and circuits in a model.

• Larger Scale Structure – It seems likely that there is a bigger picture, more abstract story that can be built on top of our understanding of features and circuits. Something like organs in anatomy or brain regions in neuroscience.
• Universality – It seems likely that many features and circuits are universal, forming across different neural networks trained on similar domains. This means that lessons learned studying one model give us footholds in future models.
• Bridging the Microscopic to the Macroscopic – We're already seeing that some microscopic, mechanistic discoveries (such as induction heads) have significant macroscopic implications. This bridge can likely be expanded as we pin down the foundations, turning our mechanistic understanding into something relevant to machine learning more broadly.
• Automated Interpretability – It seems very possible that AI automation of interpretability may help it scale to large models if all else fails (although aesthetically, we might prefer other paths).

• The End Goals – Ultimately, we hope this work can eventually contribute to safety and also reveal beautiful structure inside neural networks.

An Epistemic Foundation

Mechanistic interpretability starts by studying the "microscopic" scale of neural networks: features, parameters, circuits. It's a bottom up approach, studying small pieces without an a priori theory. This is, in a lot of ways, a very strange decision. Why not take a top-down approach targeted at the questions we care about?

Our hard won experience is that it's quite easy to be misled and confused by neural networks. If one starts with plausible assumptions about what's going on, it's often easy to find illusory supporting evidence of them if you search top-down even if one is mistaken, because there will be lots of computation which is correlated with your hypothesis.Consider a vision model trained on ImageNet. A natural hypothesis is that it will contain a dog head detector – and indeed, ImageNet models do seem to reliably contain dog head detectors! However, there will also be earlier layers, right before it contains a dog head detector, in which the model has detectors for a dog snout, dog fur, eyes, and so on, which combined can very accurately predict the presence of a dog head. This means that a linear probe will discover a "dog detector", even if the layer isn't best understood as having a dog head detector feature yet.

The critical benefit of mechanistic interpretability is that you can truly determine whether statements are true. It's an epistemic foundation:

In fact, for small enough circuits, statements about their behavior become questions of mathematical reasoning. Of course, the cost of this rigor is that statements about circuits are much smaller in scope than overall model behavior. But it seems like, with sufficient effort, statements about model behavior could be broken down into statements about circuits. If so, perhaps circuits could act as a kind of epistemic foundation for interpretability.  (Olah et al., 2020; see "Interpretability as a Natural Science")

This "foundation" is a kind of fallback that we should be able to reduce things to. That doesn't mean the ultimate plan is to work at that level of abstraction. A lot of work goes into the foundations of mathematics, but mathematicians don't do all their work in terms of foundational axioms. Computer programs can be reduced to assembly instructions, or be thought about in terms of Turing machines, but that's not how programmers work in practice. These fields have a stable foundation which can be used, with effort, to check any claim made at higher levels of abstraction, and to translate between different abstractions people build. But the foundation is just the start.

Similarly, we expect much of the exciting work of mechanistic interpretability to be building abstractions on top of the foundation we're establishing. This has been a major motivation since the beginning of mechanistic interpretability:

As we’ve studied circuits throughout InceptionV1 and other models, we’ve seen the same abstract patterns over and over. … We think it’s quite likely that studying motifs will be important in understanding the circuits of artificial neural networks. In the long run, it may be more important than the study of individual circuits. At the same time, we expect investigations of motifs to be well served by us first building up a solid foundation of well understood circuits first.  (Olah et al., 2020; see "Circuit Motifs")

If we think of interpretability as a kind of “anatomy of neural networks,” most of the circuits thread has involved studying tiny little veins – looking at the small-scale, at individual neurons and how they connect. However, there are many natural questions that the small-scale approach doesn’t address. In contrast, the most prominent abstractions in biological anatomy involve larger-scale structures: individual organs like the heart, or entire organ systems like the respiratory system. And so we wonder: is there a “respiratory system” or “heart” or “brain region” of an artificial neural network? Do neural networks have any emergent structures that we could study that are larger-scale than circuits?  (Voss et al., 2021, "Branch Specialization")

Enabling this kind of higher-level analysis to be pursued rigorously and with confidence that we aren't misunderstanding things is a central motivation of present "low-level" mechanistic interpretability research.

What Might We Build on Such a Foundation?

If our present work is building a foundation, a natural question is what can be built on top of that foundation. This section explores a few possibilities. These possibilities make us optimistic that mechanistic interpretability, despite its nature as a microscopic theory, may be able to contribute to our ultimate goals, discussed in the final section.

Larger Scale Structure

The idea that there might be "larger-scale structure" to be found once we have a solid foundation in "low-level" mechanistic interpretability isn't just speculative. The original Circuits thread was full of such findings – there were so many that it invented different categories ("motifs" vs "structural phenomena"A motif is a repeating low-level pattern within a network, while a structural phenomenon is a large-scale property. Later, we'll also discuss universality which is when a pattern repeats across different neural networks rather than within one network.) to describe several different kinds of such larger scale findings! This section will review a few of the most notable.

Unfortunately, the extensive superposition of language models has made it much harder to discover analogous structure in them, so examples will largely focus on the vision models studied in the original circuits thread. With that said, we expect that many similar things exist in transformer language models if we can get at the structure behind superposition.

Feature Families & Equivariance

Large language models contain millions of neurons. The number of features is likely much higher due to superposition. How can we hope to understand a model with such an enormous number of features? One hint is the existence of feature families.

Neural network features often appear to form families of analogous features, parameterized by some variable. This seems like one of the biggest hopes for understanding large models – even though there will be lots of features, they'll be organized. Just as a large database might have lots of entries, but still be understandable in an important sense because it's structured, so too might neural networks.

The most basic version of this is families parameterized by a simple variable like orientation, scale, or hue. This was explored at some depth in Naturally Occurring Equivariance in Neural Networks.

But it isn't just that features form these families. The circuits implementing them and the weights between them are also organized and parameterized in the same manner:

This isn't limited to simple transformations however. Neuron families can be parameterized by more abstract types of variation.

Another paper, Multimodal Neurons in Artificial Neural Networks, found even more abstract neuron families. While many of the earlier neuron families might be hard to imagine in language models, one could easily imagine families of features corresponding to prominent people, or features corresponding to emotions, in large language models.

In fact, our paper on Softmax Linear Units observed that the language models it investigated appear to have "token X in language Y" neurons, which might be seen as a simple feature family parameterized by the variables X and Y (see Section 6.3.5, "Abstract Patterns"). So the basic premise of feature families in language models seems likely to be true, at least in some cases.

The more general principle to notice here is that there can be a kind of meta-organization of features. If we step back, the reason that features are so important to understanding neural networks is that activation space has exponential volume, and we need to decompose it into features we can reason about independently to have any hope of understanding it (see this essay). But what if we use features to escape the exponential trap of activation space and it turns out there are too many features to understand? Our hope needs to be that there is also some organizational structure on the features – that we can in some sense repeat something like our trick to escape one level of abstraction higher up. Feature families are the simplest kind of organizational structure one could imagine, and it seems very widespread. This seems like cause for significant hope!

Feature Organization by Weight Structure

The feature families and equivariance we discussed in the previous discussion suggest there exists structure to be found which simplifies features – but do we have any hope of finding it buried in a mess of millions or billions of features?

One sign of hope is the results of the Branch Specialization paper. The main result is that when neural networks have branches, similar neurons (and in particular specific neuron families) will self-organize into branches. This is perhaps vaguely similar to how biological brains have "regions" that seem dedicated to certain functionality. The branches are like brain regions, and the features and circuits of a certain kind focus there. This provides a kind of automatic organization. However, that main result only works if the model has branches baked into the architecture. It also breaks down in later layers, likely due to superposition. So it isn't a very satisfying solution.

A more satisfying answer comes in a small experiment towards the end of the paper. The same organizational structure of neurons appears to be implicitly present when there aren't any branches. In fact, if anything, the situation is better than the branch version – we get a map of neurons organized on multiple axes of relationships, rather than a binary division.

This kind of "meta-organization" of features is exactly what we want. And it seems like such organizational structure should fall out of the neural network "connectome" as long as features are more closely connected to related features than unrelated ones!

It also makes sense that superposition would obscure something like this. The best superposition strategy will be to put features which are as far apart in superposition, since they'll interfere the least. But if we can undo superposition, all this structure should be there for us to discover.

Other Structure

There are hints of many other kinds of motifs and higher level structures in neural networks. Weight banding is a particularly interesting example of a larger-scale structure.

Universality

So far, we've discussed structure within networks. But it's also interesting to consider the structure which repeats across networks. We call this universality.

Universality has significant implications for interpretability because it changes the value of studying a single model. If features and circuits are universal, effort invested in understanding one model can transfer to others:

Focusing on the study of circuits, is universality really necessary? Unlike the first two claims, it wouldn’t be completely fatal to circuits research if this claim turned out to be false. But it does greatly inform what kind of research makes sense. We introduced circuits as a kind of “cellular biology of deep learning.” But imagine a world where every species had cells with a completely different set of organelles and proteins. Would it still make sense to study cells in general, or would we limit ourselves to the narrow study of a few kinds of particularly important species of cells? Similarly, imagine the study of anatomy in a world where every species of animal had a completely unrelated anatomy: would we seriously study anything other than humans and a couple domestic animals?

In the same way, the universality hypothesis determines what form of circuits research makes sense. If it was true in the strongest sense, one could imagine a kind of “periodic table of visual features” which we observe and catalogue across models. On the other hand, if it was mostly false, we would need to focus on a handful of models of particular societal importance and hope they stop changing every year. There might also be in between worlds, where some lessons transfer between models but others need to be learned from scratch.(Olah et al., 2020; see "Universality")

Many features in vision models appear to be universal, occurring consistently across a range of models. Of course, this was well known for Gabor filters for a very long time, but it also appears true for other features: