Last week, we shared more details on our vision for real time video generation. As we prepare to release increasingly powerful real-time models in the coming months, we want to ensure that we are prepared from a moderation perspective.
Why Streaming Requires Something New
With non-real-time models, a user submits a prompt (and potentially an input image or video). We scan those inputs against our safety classifiers and, if they pass, generate the output. We then scan the generation itself against our safety classifiers, and it is either blocked or presented to the user.
Streaming the generation to the user as it’s created means that the user will see the generation before we can run that final safety check. This required us to rethink our moderation system. We can still check inputs, but that process has to be much faster to meet the user’s overall latency expectations, and we have to scan the stream itself with very low latency to catch harmful content as quickly as possible. Before the video is available to download or share, we then run a final safety check to ensure the outcome doesn’t change when the full context of the video is taken into account. An individual frame of a video may seem fine but actually be problematic in the context of the full video.
Building for Synchronous Moderation
We have designed and tested a new fast moderation system that will run synchronously once a user’s input clears moderation. As the model presents the generation to the user as a live stream, we send frames of the generation to moderation. If problematic content is found, we halt the stream.
This structure requires very low latency. To achieve this, we turned to the CoPE-B model from Zentropi. This model was intentionally built to be fast without sacrificing accuracy. CoPE-B is built as a LoRA adapter over Gemma-4-26B-A4B-it: a Mixture-of-Experts model with 25.2B total parameters but only 3.8B active per forward pass. In practice, this means per-classification latency on the order of a 4B dense model, with the knowledge capacity of a much larger model.
With Zentropi, the safety review currently takes an average of less than 0.5 seconds. This means if something is missed at the input review stage, it may be shown to the user for less than a second before the stream is blocked. In our testing, we found that the most problematic content was blocked at the input review stage. Edge case inputs, which are more likely to pass the initial review, sometimes produced glimpses of prohibited content, such as adult female nipples, before the stream was blocked. The synchronous moderation through Zentropi narrows the exposure window to less than half a second.
The Layers Underneath
Real-time moderation is the newest layer; it sits on top of Runway’s existing multi-layered defenses.
Model-level safeguards. Our first layer of defense is always the model itself. We filter training data before training starts, and apply post-training techniques that teach the model to avoid generating certain categories of harmful content.
Red teaming. Before any model launch, we run extensive adversarial testing, both automated and manual. Manual testing, in particular, will focus on risks specific to a real-time model, including edge-case content that is less likely to be blocked based on input review alone.
Youth safety. We scan all user-provided inputs against known Child Sexual Abuse Material (CSAM) hash databases and a CSAM-specific classifier to detect previously unknown CSAM, as described in our approach to child safety.
Account enforcement. We use automated systems to detect problematic accounts; users who repeatedly violate our policies face automatic suspension.
Transparency. Runway’s real time model outputs will carry C2PA provenance signals so the content can be traced back to its origin, just like any other Runway output.
What Comes Next
As the real-time generation technology evolves, we will continue to iterate on our moderation infrastructure to ensure that we are minimizing the risk of potential harm while enabling creative expression for our user base.