This paper argues that no one can call recursive self-improvement safe yet, because we lack even a clear way to think about where it leads.

Recursive self-improvement, or RSI, means an AI builds a better AI, which builds a better one, and so on. AI companies are starting to pursue it, with no shared view of when it would be safe.

As an ordinary user, the author got an AI to train a small LLM from scratch from 1 high-level prompt. At a FAR.AI safety workshop, experts listed very different conditions for RSI because they imagined very different outcomes.

His 2-player game shows the RSI race is a Prisoner's Dilemma only at some risk levels. At high risk, both sides holding back is a stable outcome, so waiting together can work.