In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.
SVEET:基于双向视频扩散模型的流式视频编辑框架
AI 导读
SVEET 提出一种仅需在预训练双向视频扩散模型上训练、即可自回归实现高质量流式视频编辑的框架,核心是骨干特征解耦与条件帧独立两条原则。该方法通过辅助模型分支以时间独立自注意力编码源视频并注入骨干模块,配合解耦训练方案实现零样本知识迁移,在单张 H100 GPU 上无需辅助加速即达 15 FPS。
HuggingFace Daily Papers(社区热门论文)
42
AI 编辑部评分,满分 100SVEET:基于双向视频扩散模型的流式视频编辑框架
SVEET 提出一种仅需在预训练双向视频扩散模型上训练、即可自回归实现高质量流式视频编辑的框架,核心是骨干特征解耦与条件帧独立两条原则。该方法通过辅助模型分支以时间独立自注意力编码源视频并注入骨干模块,配合解耦训练方案实现零样本知识迁移,在单张 H100 GPU 上无需辅助加速即达 15 FPS。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org