为什么 AI 进步的瓶颈可能是人类尊严

When people discuss recursive self-improvement and what superintelligence does to the world writ large, they inevitably bring up the bottlenecks that might stop us getting near-instant leaps in GDP growth (or, perhaps equivalently, near-instant annihilation by misaligned AIs). In Machines of Loving Grace, for instance, Dario suggests these bottlenecks might be the unavoidable processes in wet labs (for biology and pharma) and physical labs (for materials science and physics); the real world has a tick rate that can’t be sped up. Zhengdong, in his 2025 letter, is less worried about bottlenecks to capability, arguing that data, algorithms, and even power have repeatedly failed to bind against compute. But he notes that we’ve ended up with ‘both the robots and the debt’: AI’s precocious cognition has met jagged diffusion, so models are more than capable of white-collar work while lawyers keep their jobs and global growth gently slows.

I wonder whether the limiting factor is none of these, but human dignity. First I want to separate two worries. One is that AI is evil and may kill everyone (which I’ll set aside here, though I’m not discounting it). The other is that, as someone put it to me the other day, AI would turn us from wolves into domesticated dogs. Framed that way, it gets a visceral reaction from a lot of people. AI researchers are one swathe of society (along with mathematicians, entrepreneurs, and the like) whose value function is largely set by their intellectual contribution to the world. More specifically, it’s their contribution over the average replacement with a similar level of intelligence and opportunity. The less likely their contribution would have happened had you swapped in the most competitive alternative, the more fulfilled they feel.

So if their most competitive replacement is always better than them (a quasi-omniscient machine intelligence), they put their fulfilment at zero, and extrapolate that out to much of humanity. At first glance this value function seems good, even noble. After all, it’s centred on helping humanity as much as possible. But think about it for a bit and it’s deeply anti-altruistic (I don’t mean that normatively; you can have whatever value function you want). It says you’d rather everyone be worse off and you still make the biggest contribution than live in a world where everyone’s utility is much higher and your contributions are superfluous. It is remarkably prescient of Silicon Valley’s Gavin Belson, the CEO of Hooli, to say “I don’t want to live in a world where someone else makes the world a better place better than we do.”

And when I think about who holds this view, it’s the people deciding the future of the lightcone. The powerful AI researchers and entrepreneurs building this technology are the ones most likely to feel that being a domesticated dog isn’t worth it. (Plenty of others will feel the same, mathematicians for instance, but they have far less control over how AI goes.) Admittedly some of these researchers and entrepreneurs won’t feel this way, because their value function looks more like max(contribution) over time than cumsum(contribution). For them it will be enough to have ushered in RSI, made a lot of money, and known they played a big part in birthing a new intelligence; one big contribution rather than an ongoing one.

I think even this group is quite small, though. What matters to most of these people is just knowing they’re capable of making the best contribution over any replacement. And many of them would feel differently if they were actually faced with a world where compute is taken off them and handed to the very top humans and increasingly intelligent models. Really feeling the obsolescence might turn a max into a cumsum. Nor do I think many of them will care that a lot of people, perhaps most of the workforce, would be happy spending far more time with friends and family and having a much higher quality of life, without having to work. Lack of meaning affects everyone, of course, but there are plenty of people who, at least explicitly, wouldn’t mind retiring tomorrow.

So I can imagine a world where we handicap the AI to some extent. We don’t let it spend quadrillions of tokens developing new areas of mathematics; we spend those tokens on having it explain the solutions it found to the problems we posed, so we can pose the next one. The same people will argue that no matter how intelligent the machine gets, we still need a human in the loop for the non-verifiable problems of distribution of resources, of what our legal system should look like and how to judge right and wrong, how to assign finite compute to infinite human wants, needs and desires (how much to give to curing cancer vs procedurally generating AI video games on the fly for people), ethical issues such as cloning or whether we should keep people alive forever if we can, and so on. The system will evolve to protect those powerful people’s voices, so at least they can feel like they are influencing a now no-longer capitalist system because the most intelligent way to allocate resources is not being used. (Again, I’m not saying whether that’s good or bad, just guessing at what might happen.)

There aren’t many bets I’d take against scaling curves and intelligence, but one of them is the human ego. That is an incredibly powerful scaling curve in its own right.