Looped transformers are a popular architecture topic right now.

This new technical report extends the loop across tokens.

Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included.

A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations.

With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same.

The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states.

The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet.

Paper: https://github.com/yifanzhang-pro/recurrent-looped-tranformer

Chat with Paper: https://academy.dair.ai/papers/recurrent-looped-transformer