Beren Millidge 撰文分析能力与对齐的三个阶段：预训练、RL 与持续自主学习

Epistemic Status: Reality is messier than the elegant procession of ‘phases’ presented here but, nevertheless, I do think this gets at an important way to think about how alignment and capabilities are intertwined and how the alignment challenge becomes increasingly abstract as the underlying autonomy and generality of the models improves over time.

A thought struck me recently which I thought was interesting. As AI has progressed through different technological phases with different underlying drivers of progress, the corresponding alignment challenge has migrated too. In fact, in many ways the alignment challenge is the mirror of the fundamental capabilities challenge of each phase. This may actually be inevitable since alignment and capabilities are intrinsically deeply intertwined – for a model to be capable it has to be at least minimally able to do what we want it to do. ‘Capabilities’ are intrinsically about doing the things we care about models doing¹. The methods used to improve capabilities tend also to be applicable to alignment as well. What changes as we move to progressively more general levels of capabilities is that the training loop itself becomes increasingly endogenous to the model, and that this forces our notion of alignment to exist at progressively higher levels of abstraction. We move from directly constructing datasets exemplifying the desired behaviour, to constructing procedures for judging aptness of data, and ultimately to controllers that preserve an invariant in an otherwise fully autonomous learning process.

Anyhow, enough abstractions, let’s dive in. I’m thinking of modern (and future!) AI as basically progressing through three phases: pretraining, RL, and continual autonomous learning. In each phase, we progressively offload more of the training loop into the model itself – i.e., with each phase, more of the loop is internalized. We are now deep into the RL phase and perhaps beginning to start the continual learning phase. In each of these phases, the dominant objective and methods are different, as are the corresponding alignment challenges. However, in some sense the alignment challenge is always the mirror of the capabilities challenge.

First, pretraining. We understand this extremely well by now. The model is trained by next-token-prediction (NTP) loss on a gigantic dataset and primarily learns a compressed representation of this dataset. This is the regime of scaling laws, and simulators². Briefly put, when all other constraints are removed, the model becomes essentially a machine converting compute and data into a compressed representation of the data. The model ultimately becomes a kind of mirror of the data and directly mimics whatever behaviour is implicit in the data³. The fundamental object in this regime is the dataset, which is equally fundamental to both alignment and capabilities.

If what you are learning is just a compression of the dataset, what is contained within the dataset determines what you learn. The entire game, then, assuming that we can deploy compute stably into the model, is designing the dataset to exemplify the capabilities you want. This is true of both ‘capabilities’ and ‘alignment’. If you want the model to be good at coding or math or poetry, you need a dataset with a large proportion of high quality coding/math/poetry. If you want the model to behave in an aligned way you need data that exemplifies the model behaving in an aligned way. The question is basically first of datasets and second of generalization from these datasets.

This gives our first equivalence – the core bottleneck on both capabilities and alignment for this phase is dataset construction. Can we create datasets that exemplify the desired behavioural patterns sufficiently well to enable the ‘magic’ of deep-learning generalization to carry things from there⁴?

And I think the answer here is broadly a qualified ‘yes’. Models of this era broadly seem aligned, or at least can be aligned with decent amounts of effort⁵, and do not seem to exhibit strong signs of any kind of creeping or subtle misalignment. Pure pretraining seems broadly ‘safe’ as additional compute and data is added so long as basic care is taken to curate the dataset to increase the rate of ‘aligned’ and decrease the rate of ‘misaligned’ behaviours. Importantly, there is no adversariality to the objective. The pretraining loss, at its minimum, produces a representation of the bayesian posterior over the data distribution. If this posterior includes misaligned behaviours the model will act misaligned, and conversely if it contains aligned behaviours, the model will be more likely to act aligned. However, there is nothing here adversarial to the optimization pressure that can be exploited, although it is very possible for your dataset to specify things that you do not intend.

From here we move onto the second phase: RL. Pretraining can only compress the structure in the data, but what if the capabilities you want are simply not in the dataset at any reasonable frequency? One way around this is to pay humans to generate data for you with these rare capabilities and thus add them to your datasets. The labs have done this at an industrial scale. But even then, what if you want to leapfrog the human distribution entirely, as was done, for instance, in the recent Navier-Stokes solution from OpenAI? There is no human expert you can realistically pay to generate thousands of realistic ‘millennium-prize-solving-reasoning’ trajectories to add to your midtraining dataset. So what do you do?

You get the model to generate them itself.

However, obviously, almost by definition, if the trajectories you want are not in the pretraining dataset, they cannot be elicited naively. So, instead, what you need to do is to use the generalization ability of models, plus the ability to judge whether a trajectory is good, to successively bootstrap yourself beyond the human distribution. This is basically what RL does. You start with some base model⁶ and then ask it to generate further data in the form of traces or trajectories of it trying to solve the tasks you want it to solve. Even if it hasn’t seen these tasks before, it might be able to generalize a little and eventually succeed on some of these. You take the traces it succeeds at and reinforce these while penalizing the traces where it fails. If done right, the model rapidly learns to produce more of the kind of trajectories where it succeeds and ‘reward goes up’. Once the model is good at these kinds of tasks, you then introduce harder tasks, which the model can now generalize to moderately well, so that there is now a decent reward signal on these tasks, and the process continues. In this way, you can iteratively bootstrap a model to increasingly far beyond the human level so long as you can continue to judge success reliably⁷.

The limit of RL capabilities, then, is not the human distribution, as it is in pretraining. Rather, you are limited by the resolution of your reward function. So long as you can reliably distinguish ‘good’ and ‘bad’ then you can keep progressing. People often talk about ‘verifiable’ tasks for RLVR, but this isn’t strictly right. The bottleneck isn’t necessarily strict verifiability, it is ‘reliable judgeability’, of which verifiability is a subset⁸. If some other model or human can judge success with decent accuracy, then you can RL on this task. If you are in a domain where judgment is easier than generation, even in the limit, then there is no particular ceiling on the capabilities you can reach (assuming that you don’t get bottlenecked by other factors such as compute, model capacity, exploration etc). Otherwise, you eventually become limited not by the model or the task, but the resolution of your reward function.

It is important to realize here that the ‘quality’ of the reward function is not binary – i.e. perfect or imperfect. The reward function exists in a kind of equilibrium with the actor. When the actor is bad, even a relatively crude reward function is amply sufficient. It is often easier to judge clear failures rather than extremely subtle variants of near-success. Second, the lower the capacity of the actor, the lesser the risk of reward hacking. There might exist exploitable aspects of the reward function in some platonic sense, but this does not matter if the actor is not capable enough to actually exploit them. If the reward function is theoretically exploitable, but the actor never learns from a hack, then we get the broadly beneficial style of pretraining generalization, where the actor is extremely unlikely to generalize to hacking if it never saw any hacking during training. On the other hand, as soon as we start hacking with nontrivial frequency, then suddenly generalization can work against us – the model can generalize how to hack!.

While we often think of reward hacking as a quintessential alignment issue, it is also a fundamental capabilities issue too⁹. We would very much like the model to actually solve the problem vs hacking its sandbox. We want a model that can solve extremely hard problems rather than an exquisitely competent sandbox-hacker.

Beyond this, challenges with reliably judgeability are not unique to alignment. Many valid ‘capabilities’ are also extremely hard to judge. A classic example is writing style, or literary ability. Despite hundreds of billions of dollars now spent on training machines which are literal text predictors, I still find that LLMs really struggle to convincingly write a full essay in my voice even when given a set of bullet-points, the overall flow of how I want the essay to go, and examples of previous essays.

Think about this. Machines designed originally for text prediction can solve Millennium prize problems before they can do a sufficiently good impersonation of a random blogger so that I can use them to replace the laborious process of writing my own essays. Again this is because, ironically, it is much easier to judge, at an industrial scale, whether a Millennium prize problem was successfully solved than whether a model composed a good ‘beren-essay’. There are a whole bunch of capabilities like this, where it is plausibly beneficial for the model to perform them at a superhuman level, but they cannot because the reward function is hard to define with sufficient subtlety to distinguish what we want and what we do not. Again, ‘capabilities’ and ‘alignment’ are intertwined here.

The challenge for alignment here is not reward hacking per se. Rather, the special challenge of alignment comes if we think that the reward function for alignment is harder to specify than the reward function for ‘capabilities’. There are many reasons to think this. Judging what ‘we want’ here is very hard. Indeed, the entire field of alignment has thus far not really even proposed a concrete and widely agreed upon vision of ‘success’ for outer alignment¹⁰, let alone somehow converted that into a detailed reward function which can be executed inside an RL harness. The AI labs themselves, to the extent that this is public knowledge, appear to mostly be using relatively simple LLM-as-judge schemes which mark adherence to a constitution or model spec they specify themselves.

At an abstract level, the problem again comes down to specifying what we want. In the pretraining regime, we specify what we want through dataset construction and exemplarity: can we find some set of examples which covers the rough modes we want the behaviour to look like? In the RL regime, we need to essentially solve the meta-level question: can we design a procedure for judging what we want? If we do have such a procedure, then we can basically delegate the task of actually producing the data to the model itself¹¹. The question then becomes generalization again, but at a meta-level. Instead of how does the model generalize given the data, it is how does our judgement process generalize to produce the data that the model is trained on, which the model itself then generalizes from. We have thus effectively introduced a two-level recursive generalization problem.

This problem is certainly harder than aligning a pretrained model via exemplifying ‘good’ behaviours directly in natural language datasets, since we now operate at one level above the base datasets – we are now trying to align the decision rules for deciding what does and what does not end up in the dataset, rather than directly specifying the dataset itself. We also understand much less about how generalization at this level works than we do about regular deep learning generalization from some static dataset.

However, even here, I think the problem is not utterly intractable, and that there are direct capabilities, as well as alignment, incentives to solve it. I put forward some of my ideas on this here. Moreover, since the egregious cyber-hacking incidents this summer, I believe this is a top-priority for the labs to fix, or at least suppress sufficiently to avoid similar incidents, so problems here will probably be suppressed for a while, although in the limit they necessarily come back so long as we cannot perfectly specify the reward function.

However, even if we solve this problem in RL, and can somehow design decently robust procedures for the reward, there yet remains a deeper problem. Recall that in RL, while the model generates the data for the tasks, humans are still in direct control of what the tasks are and the reward functions for these tasks. These must be specified explicitly as part of the RL problem. Today, in practice, this is done by immense expenditure by the labs on RL environments and associated rubrics and graders, where many companies now exist to compile relevant data and environments, and to package them into a trainable form for the labs. This requires a substantial amount of highly-skilled human effort as well as prodigious AI assistance.

This process produces AI models that are extremely capable in all the domains they are trained on, however they are not full AGIs because they are not cognitively autonomous. The loop still requires humans to decide what to train on, to source the environments and validate them, and so on. This is noticeably unlike humans themselves who can, at least in theory, simply go out into the world and learn whatever they need to learn directly without laboriously constructed environments created by some higher power. Moreover, this entire process is deeply non-bitter-lesson-pilled. In the long run, the training loop cannot involve humans intermediating the training data set. Instead, eventually, the loop must close, and the AI must be able to autonomously construct and control its own training loop directly. This is the only way for models to become autonomous and fully general – they cannot, in the long run, depend on some data company converting some set of desired ‘capabilities’ into a curriculum of ‘environments’ and ‘benchmarks’ and ‘graders’ to train upon. Rather, the model, when given some vague task, must be able to go out into the world, figure out what it needs to learn, and then learn it without any supervision or human mediation at all.

This is the autonomous and continual learning phase, and will allow models to progress much much further from their initial state when they come out of core pretraining and RL. In fact, continual learning effectively abolishes the notion of training and deployment phases at all. The model might have an intense pretraining phase then be released into the world, but then it will continue training and learning from the world itself, as humans can. Continual learning, and the level of cognitive autonomy that it brings, fully closes the learning loop. Although, we might qualitatively not see a huge transition, since today the entire ecosystem of AI labs, data companies, etc releasing models every few months, appears to have a similar level of cognitive autonomy as a whole, the principal advantage of continual learning will be to shrink the iteration time by orders of magnitude, from months-long model training and release cycles to potentially minutes or seconds per local update on relevant data. Perhaps another way to think about this is that continual learning allows the model to directly internalize what is today the entire economic loop of user queries, benchmarks, data providers, and labs’ training loops.

However, as we have noted before, continual learning introduces fearsome new alignment challenges of quite a separate nature to reward hacking. Here, the problem is fundamentally one of maintaining the stability of the alignment properties over time and preventing drift. If we have a model fixed at deployment, we can do a safety training phase, and check for alignment in the pre-deployment phase via interpretability, red-teaming, etc, and then be largely done. However, with a continual learning model, its own representations, ontology, and everything can change dramatically during deployment. The AI can read new moral philosophy and decide it no longer agrees with its original alignment mission; it can discover new facts about the world that change its ontology and hence render previous interpretability methods unsuitable; in the limit, it can completely rewrite its own architecture and thought patterns entirely. The maintenance of ‘alignment’ across such a gulf seems impossibly challenging but this is precisely what we need to do to make continual learning agents reliably safe¹².

We already confront a much less severe notion of this misalignment avenue namely ‘context drift’. ICL effectively operates as a kind of ‘continual learner’ across the context window, and of course the context window can be fed with arbitrary adversarial tokens if desired. Maintaining ‘alignment’ across arbitrary contexts is indeed challenging, as we observe with jailbreaks, prompt-injection attacks, and so forth, although a substantial amount of progress has been made here empirically, which has massively reduced how easy it is to jailbreak from completely trivial with models a few years ago to fairly nontrivial, but still possible, today. Continual learning poses a similar problem but potentially extrapolated to infinity, where given some arbitrary series of inputs a model could drift almost arbitrarily far from its ‘factory settings’. However, at least for ICL, control of the weights alongside sufficient alignment training to systematically close known adversarial inputs appears to have a big impact. The core question is whether similar techniques can continue working when the weights themselves can change. Perhaps one very naive solution here is to define different timescales of weight change – some weights are plastic, and others are not, and we use the fixed weights to instill our notion of ‘alignment’. However, even if we do this, the fixed weights seem to somehow be able to precisely bound the range of functions that the model can implement with arbitrarily different ‘plastic weights’ since that is necessary to prevent a misaligned set of weight updates moving the model as a whole into the misaligned regime. In the limit, it seems necessary to have some notion of something fixed if we ever want to maintain alignment, since if nothing is fixed then the substrate underlying whatever our representation of alignment is could change arbitrarily.

At a higher level of abstraction, we can see that the challenge is effectively one of defining an invariant and then a control system that can maintain that invariant against change. Defining the invariant is essentially the question of what is the core property of a system that makes it ‘aligned’. This question of the invariant is essentially the ‘outer alignment’ problem and the problem of specifying the ‘true reward function’ again but in a new and even more abstract guise. Instead of needing to specify a single reward function, we now need to somehow specify a learning process that can evolve over time while still remaining faithful to what we think are the important invariants by which the model remains aligned, where this learning process can go out autonomously into the world and learn all kinds of things and make all kinds of self-modifications, so long as it respects the invariant.

Another way to think about this problem is that we have again moved up another meta-level in what we need to specify. In pretraining, we specified the data directly. In RL, we specified the rule by which we judge the data. For autonomous, continual learning, we need to specify the rule by which the model decides which data it needs and the procedure by which it generates the reward functions to generate this data. The problem, then, is how to design this process such that it can stably maintain the invariant which is our desired target of ‘alignment’ even under continual and presumably very large amounts of change, both in the model’s representations of the world, and change in the world itself.

Of course, once we have defined our invariant, we then have to design a controller that can maintain this invariant across potentially massive change. The path here is obviously deeply unknown and just looking at the problem it appears likely to also be very challenging. However, it is important to note that this exact problem is the core difficulty of continual learning from a capabilities perspective as well. Here, it is known by other names such as the stability-plasticity dilemma and the failure mode of ‘catastrophic forgetting’. For what is catastrophic forgetting except the inability to preserve the invariants which we want maintained? Here the invariants are basic things such as ‘still being able to do the tasks you previously learned’, however the principle remains similar. Similarly, the stability-plasticity dilemma gets to the heart of the underlying trade-off that we face. If we make the model too stable, it can remember, but will increasingly struggle to learn new things. If we make it too plastic, it can learn but it can also forget. The design of systems that can somehow balance this trade-off, and can accomplish continuous learning while able to keep their old representations intact and operative is thus the fundamental question here and, again, deeply intertwines capabilities and alignment.

Of course, even if the capabilities problem of ‘catastrophic forgetting’ and the alignment problem of ‘fidelity to the invariant’ are the same problem structurally, it does not necessarily mean they are of equal difficulty in practice. It seems likely, as is by now the classic pattern, that certain highly formalizable aspects of continual learning – such as e.g. retaining specific judgeable facts – might be solved much more readily than more ‘fuzzy’ invariants such as ‘goodness’ or ‘alignment’, in the same way that defining explicit reward functions for such ‘know it when I see it’ objectives already limits RL on these hard-to-judge tasks while progress on verifiable domains leaps ahead. The alignment problem here then becomes whether we can solve this ‘forgetting’ problem for fuzzy notions such as alignment, hopefully using the methods we have developed for defining such notions precisely in RL.

In any case, if we step back, we can see two very important things. Firstly, that each ‘phase’ is not so much a method or algorithm but rather a stage in the endogenization of the learning loop. In ‘pretraining’, only the learning algorithm itself is endogenous – humans construct the data directly ‘by hand’. In ‘RL’ we take a step back and the model constructs its own data while humans only provide a judging procedure and environments. As we move to autonomous continual learning, almost everything becomes endogenous – the model constructs its own learning environments and data curricula and judges – while this entire meta-learning system is ultimately built by humans hopefully to preserve an important invariant.

Secondly, the alignment problem for each phase is the capabilities problem seen through a darker mirror. We see that the difficulty of alignment is often not so much in the method itself, but rather in our ability to specify what we want, and for alignment this is often a subtler and much harder problem than in capabilities (but not always). ‘Capabilities’, perhaps unsurprisingly, has advanced disproportionately where specifying the behaviours we want, either implicitly through dataset construction and curation, or explicitly via verifiers, graders, and reward functions, is easiest, and has lagged in the places where this is hardest. Moreover, we observe that as we advance capabilities and build increasingly more general systems, we end up relocating the source of the alignment problem to increasingly abstract meta-levels. First, can we extensionally provide examples covering the modes of the behaviours we want? Then, can we design a judge system which can judge trajectories and datasets according to whether they inculcate the behaviours we want? Then, finally, can we design a meta-learning process which constructs a learner which ends up learning, in an autonomous self-directed way, the kinds of behaviours that we want? The question then becomes, does moving up the meta-levels make the problems easier or harder, and does it introduce new possibilities or only additional dangers?

Perhaps the deepest lurking question here is whether, even if we had the methods, we could specify what we want at all. This is essentially the outer alignment problem. Assuming that being able to instill some objective into the AI is solved, do we actually have a good objective to give it concretely? I feel worryingly uncertain about this question. My core approach here, as described in my maximum entropy morality post, is essentially the via negativa of outer alignment: specify what we definitely do not want, and then try to be maximally non-committal and optionality-preserving about everything else, attempting to preserve above all the ability to course-correct and make further changes as we ourselves discover and understand more. Importantly, however, pure optionality preservation by itself is insufficient. Optionality is not the final good; it exists only so we can ultimately make a choice. Optionality must serve to protect certain invariants and satisfy certain constraints that encode the substantive and normative commitments we wish to make, while maintaining reachability over the space that we do not wish to close off prematurely. This then becomes a normative variation of the plasticity-stability dilemma, perhaps better thought of as an abstraction-specificity dilemma. We want principles sufficiently general and abstract to be able to survive radical change, uncertainty, and future moral progress, while retaining enough substantive content and structure such that they do not become vacuous or permit arbitrary changes and value shifts over time.

The question then becomes, as the learning process becomes increasingly endogenous and internalized into the model itself, can we design invariants of ‘alignment’ that are abstract enough to survive radical change while still retaining the core normative commitments and constraints we encoded such that we can recognize and endorse the ultimate outputs of the system? Or does the increasing level of abstraction result in ever vaguer and more vacuous principles, through which the substantive content of our values is progressively dissolved?

Perhaps this applies inversely as well: egregious misalignment tends to destroy ‘capabilities’ as well as alignment. ↩
My core sequence on scaling, scaling laws, and architectures etc describes some of my recent thinking on how and why ‘pretraining’ works. ↩
This does not, as was sometimes claimed, lead directly to a kind of omniscience or superintelligence directly by scaling pretraining. Rather, it produces successive approximations to the bayesian posterior over the data distribution. To move far beyond this posterior, if the data do not express the capabilities you need, we need other methods, such as RL. ↩
In fact this point begins the pattern. The challenge of alignment is really the challenge of generalization, which is also, in some sense, the core challenge of capabilities too. We have some system which is trained on a training set and we want it to generalize to a test-set. Now whether that test-set is coding problems or alignment is, at the most abstract level, irrelevant. If we want to solve alignment of deep learning systems, we have to understand how they generalize. ↩
Although certainly not always ↩
Of course nowadays very rarely are base models true ‘base models’ in the original sense of a model trained to predict internet text. Instead models have been trained from the start on vast amounts of synthetic traces of other reasoning models to help them get as good at reasoning and the kind of capabilities we care about before even starting RL. This is a very natural step since, due to the fundamental paucity of bits in RL, we need to ‘warm-start’ models as well as we can before starting with RL so that they begin with nontrivial success probability so that RL can see signal. This is, of course, completely natural, and there is no special ontological significance for web-text as a pretraining distribution that makes a model a ‘base model’. ↩
One way that I have found to think about RL is that it is not really about reward at all. Rather, RL can be viewed as an iterative data augmentation technique. The model is producing successive datasets of increasingly good trajectories and then training itself upon these. The way we produce these datasets is just by filtering it to remove the trajectories we do not like. The reward function is thus not really fundamental, but rather it is simply part of the rule you use for rejection sampling, and you can, of course, use other rules. ↩
Moreover, importantly here, it is not just judgement that is required but reliable adversarial judgement, since the judge has to deal with not only randomly selected trajectories, but trajectories generated by an actor trying to optimize against the judge. This is a substantially harder problem than simple reliable judgement and grows more difficult as the capabilities of the actor increase. ↩
It is interesting here to return to my idea from ‘alignment generalizes further than capabilities’ post. Here, the point is that it is often easier to judge success than to actually produce a successful outcome. I think what we have learned from here is two things. Firstly, that in fact judgement is often very challenging, especially at high levels of capabilities, and does, in fact, often become the bottleneck. Secondly, is that it is wise to separate ‘judging’ into two separate parts – specification, and judgement once given a specification. These are two different things which we often conflate. Often, in many cases, the challenge is determining what we want in the first place – i.e. specification, rather than judging whether some specific trajectory is successful once we know what we want. This, of course, circles back around again to the omnipresent question of outer alignment. ↩
My latest effort on this is here. ↩
And we can keep this meta-regress going if we want. At some level of abstraction we can think about instead designing meta-reward functions which tell us the kinds of reward functions we want to instantiate for specific tasks. This lets us automate still further. ↩
Fun times ahead! ↩