We believe instant generation is where video models are headed. Last week, we released Solaris and GWM Worlds 2, and detailed our approach to agent training in digital and physical worlds. Today, we’re sharing a broader look at our research efforts in real-time video generation.
Today’s video models mostly work in distinct stages, with each output existing as a single finished object. You prompt, you wait, you get a video. Our work in real-time generation is taking a different approach, with a focus on optimizing for time-to-first-frame (and subsequently, streaming video as you prompt it); we first discussed this when we released Runway Characters earlier this year.
We’ve spent nearly a decade building tooling for training foundation models from scratch, and we use this tooling across our frontier model development. The same infrastructure and training optimizations that have driven our work on video models in the past have been critical to accelerating our research on real-time generation.
Why Real Time
Breaking down the barrier that still exists between ideation and creation will bring a level of speed and control to video generation that current models can’t match. And when models run faster, they take up less GPU time, making them more cost effective.
Today, video generation is relatively static: you input a prompt, wait a few seconds or minutes, then get a finalized output. If the clip you’ve generated doesn’t match your vision, you need to start from scratch. Over and over, we hear that generation and iteration time is the slowest part of our users' process. With instant feedback, the bulk of time can switch to actively directing, rather than passively waiting.
Decreasing time to first frame (and overall generation time) also changes the economics of creating with video. We consistently find that cost per output at a given quality bar determines viable use cases. Instant generation pushes that bar out further, bringing use cases that weren’t previously cost-effective into reach.
How it Works
Our approach to real-time video models focuses on post-training our base foundation models, such as Gen-4.5. We condition each autoregressive step on two things: 1) an initial first frame and 2) a caption, which the user chooses as input at the start. You can then generate frame-by-frame, keeping all generated latents in context in order to preserve quality. The video and audio decoders run causally, streaming the outputs as latents are generated.
Research
Getting the Model to Run in Real-Time
To make our base models run in real-time, we first make their generations causal and autoregressive, then distill the teacher into a high quality real-time model.
Teacher forcing: making it causal. We convert the entire model architecture into a temporally causal, frame-by-frame autoregressive generator. While we’ve consistently released industry-leading models focused on full-attention generation, this stage is evidence that the same architecture adapts cleanly to an autoregressive setting without losing what made it strong in the first place.
Student forcing: speeding it up. The causal model from stage one is still slow: flow matching takes many denoising steps to produce a frame. Distribution matching distillation gets each frame down to a few steps, fast enough to stream with good latency. This distillation runs in two stages: first off-policy, then on-policy. Off-policy distillation provides a strong starting point; most of the actual gains in the generation pipeline come from the on-policy distillation process.
Off-policy. The student causally predicts next states given ground-truth context. A frozen bidirectional teacher model shows the student what a good generation looks like, and a critic model tracks what the student currently produces, pushing the student toward the teacher's distribution. The distribution matching objective thus closes the gap between what the student predicts and what the teacher predicts.
We denote this training paradigm as off-policy as the model does not perform rollouts and does not observe its past actions. This keeps training memory and step time low. It's an efficient way to start, but it isn't sufficient by itself as error accumulation leads the model out of distribution very quickly in the video domain. Unlike with language models, which can correct themselves midstream via text, a small error in video generation compounds over time, as the model builds on each frame sequentially.
On-policy. Because artifacts generated in one frame of a video snowball to become large morphs later on, we use on-policy distillation to reduce error accumulation. In on-policy distillation, we perform rollouts inside training itself: the student generates a sequence autoregressively, with each generated latent becoming the context for the next one. The model now sees its own generated context in training, which helps it correct its own drift instead of amplifying it. This formulation is now equivalent to inference, where the model inevitably sees its own mistakes.
We find that training on an increasing-sequence-length curriculum outperforms having a fixed sequence length throughout. Starting training with long sequences immediately causes generated trajectories to diverge sharply from the teacher’s guidance at the end. As errors accumulate across the trajectory, the model drifts toward out-of-distribution states; by the end of a long rollout, the teacher’s supervision targets an outcome completely disjointed from the student’s actual path, rendering the training signal unstable and inconsistent.
Infrastructure: Serving Real-Time Video
Real-time generation moves the compute bottleneck from training to inference: every frame has to leave the model fast enough to keep up with playback, on hardware that's shared across concurrent sessions.
We’re building and evaluating our inference stack the way we evaluate every production system at Runway: using our internal eval and observability tooling to compare generation quality and latency across checkpoints and hardware configurations side by side. We’ve written extensively in the past about our infrastructure, training and inference optimization work, and we’re building all of this into our real-time work.
Where This Goes Next
Instant generation is foundational to interactive experiences, which we think will ultimately be the biggest use case for generative media. Education, gaming, robotics and beyond will all depend on video that responds as fast as the person watching it. The same holds for simulation: evaluating how physical agents (robots, autonomous vehicles, etc.) act in the world requires environments that generate in real time and respond to edge cases instantly; we believe the same holds true for digital agents to perform complex computer use tasks across every online space. This belief is driving much of our foundational research today.