LLaDA2.0-Uni：以扩散大语言模型统一多模态理解与生成

   

   

AGI Research Center, Inclusion AI

🔥 新闻

• [2026-05-29] 📣 SGLang Omni 支持已就绪。安装与使用请参见 cookbook。

• [2026-05-12] 🖥️ 我们发布了 ComfyUI 与 Diffusers 支持。安装与使用请参见 apps。

• [2026-05-06]⚡ 我们发布了FP8 量化版本在HuggingFace和ModelScope.

• [2026-04-23] 🎉 我们发布了 LLada2.0-Uni 的初始版本，包括：

  • 🎯 模型检查点已上线 HuggingFace！
  • 🎯 文生图（带思考模式）推理代码！
  • 🎯 图像理解推理代码！
  • 🎯 图像编辑推理代码！
  • 🎯 面向 dLLM 骨干网络的 SPRINT 加速！

📝 待办事项

• 量化模型
• Diffusers 支持
• ComfyUI 支持
• SGLang 支持
• RL 优化

📚 模型介绍

我们推出 LLaDA2.0-Uni，这是一个基于 dLLM 的统一 Mixture-of-Experts（MoE）模型，将多模态理解与生成无缝集成于一体。

架构创新

• 统一 dLLM-MoE 骨干网络：基于 LLaDA 2.0 构建，将多模态理解与生成统一为简洁的 Mask Token Prediction 范式。

• 离散语义 Tokenizer：利用 SigLIP-VQ 将视觉输入转换为离散语义 token，显著增强多模态理解能力。

• 高效扩散解码器：将离散 token 与专用扩散解码器配对，实现高保真生成，并通过知识蒸馏实现快速的 8 步推理。

核心能力

• 顶尖的理解与生成能力：在回答视觉问题和理解文档方面可与专用 VLM 媲美，同时还能生成高度精细的图像。

• 灵活的图像编辑：支持单参考或多参考编辑。可实现精确修改，同时完美保留原始细节。

• 交错生成与推理：借助统一的离散表示，轻松处理复杂的交错生成，并解锁先进的交错推理能力。

📊 评测结果

📌 快速开始

⚙️ 安装

1. 创建 conda 环境

git clone https://github.com/inclusionAI/LLaDA2-Uni && cd LLaDA2-Uni
conda create -n llada2_uni python=3.10 -y
conda activate llada2_uni

2. 安装 PyTorch（CUDA 12.4）

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

3. 安装 Flash Attention 2（高效推理所需）

pip install flash-attn --no-build-isolation

4. 安装其余依赖项

pip install -r requirements.txt

🧨 推理

🌟 文生图生成

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from decoder import decode_vq_tokens

model_path = "inclusionAI/LLaDA2.0-Uni"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="cuda", torch_dtype="bfloat16", trust_remote_code=True
).eval()
model.tokenizer = tokenizer

# Generate image tokens
result = model.generate_image(
    "A modern Scandinavian kitchen with white cabinetry, marble countertops, and a single orchid on the island. A Nordic woman with sleek blonde ponytail, wearing an oversized sweater and dainty silver necklaces, stirs a matcha bowl with a bamboo whisk, eyes sparkling with quiet joy. Shot with 50mm, f/2.5, diffused window light, cool white balance, low saturation, clean skin retouch. Mood: serene, wholesome, hygge.",
    image_h=1024, image_w=1024,
    steps=8, cfg_scale=2.0,
)

# Decode to PIL image (default: 50-step ODE)
image = decode_vq_tokens(result["token_ids"], result["h"], result["w"], model_path, "cuda")
image.save("output.png")

[!Note] 💡 更快的解码 — 使用 decoder-turbo（蒸馏解码器）可实现 约 10 倍速的图像解码（8 步而非 50 步），且质量损失极小：

image = decode_vq_tokens(
    result["token_ids"], result["h"], result["w"], model_path, "cuda",
    num_steps=8, decode_mode="decoder-turbo",
)

🌟 带思考的文生图生成

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from decoder import decode_vq_tokens

model_path = "inclusionAI/LLaDA2.0-Uni"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="cuda", torch_dtype="bfloat16", trust_remote_code=True
).eval()
model.tokenizer = tokenizer

# Generate image tokens with thinking process
result = model.generate_image(
    "A fox with thick, dense, fluffy fur in a winter setting, possibly surrounded by snow.",
    image_h=1024, image_w=1024,
    mode="thinking",
    steps=8, cfg_scale=2.0,
    thinking_steps=32, thinking_gen_length=4096,
)

# Print thinking trace
print("Thinking:", result["thinking"])

# Decode to PIL image
image = decode_vq_tokens(result["token_ids"], result["h"], result["w"], model_path, "cuda", num_steps=8, decode_mode="decoder-turbo",)
image.save("output_thinking.png")

🌟 图像理解

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from encoder.image_tokenizer import ImageTokenizer
from decoder.smart_img_process import smart_resize_images

model_path = "inclusionAI/LLaDA2.0-Uni"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="cuda", torch_dtype="bfloat16", trust_remote_code=True
).eval()
model.tokenizer = tokenizer

# Encode image to discrete tokens
image_tokenizer = ImageTokenizer(model_path=model_path, device="cuda")
pil_image = smart_resize_images(["./assets/understanding_example.png"])[0]
info = image_tokenizer.encode_with_info(pil_image)
image_tokens = [x + model.config.image_token_offset for x in info["token_ids"]]
_, h, w = info["grid_thw"]

# Understand the image
response = model.understand_image(
    image_tokens, h, w,
    question="Describe this image in detail.",
    steps=32, gen_length=2048,
)
print(response)

🌟 图像编辑

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from encoder.image_tokenizer import ImageTokenizer
from decoder.utils import generate_crop_size_list, var_center_crop
from decoder import decode_vq_tokens
from PIL import Image

model_path = "inclusionAI/LLaDA2.0-Uni"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="cuda", torch_dtype="bfloat16", trust_remote_code=True
).eval()
model.tokenizer = tokenizer

# Encode source image
image_tokenizer = ImageTokenizer(model_path=model_path, device="cuda")
crop_size_list = generate_crop_size_list((512 // 32) ** 2, 32)
pil_image = var_center_crop(Image.open("./assets/edit_example.png").convert("RGB"), crop_size_list=crop_size_list)
info = image_tokenizer.encode_with_info(pil_image)
image_tokens = [x + model.config.image_token_offset for x in info["token_ids"]]
_, h, w = info["grid_thw"]

# Edit the image
result = model.edit_image(
    image_tokens, h, w,
    instruction="Change the background to a beach.",
    steps=8, cfg_text_scale=4.0,
)

# Decode to PIL image
edited_image = decode_vq_tokens(result["token_ids"], result["h"], result["w"], model_path, "cuda", num_steps=8, decode_mode="decoder-turbo",)
edited_image.save("edited.png")

🌟 SPRINT 加速

SPRINT 通过结合 KV cache 复用、自适应解掩码 和 基于阈值的批量接受 来加速推理：

• KV Cache 复用与剪枝：前缀 KV cache 在预热步骤中计算一次，随后可选地根据重要性分数（将 KV 注意力重要性与 token 置信度混合）进行剪枝。后续去噪步骤复用缓存的前缀，显著减少计算量。按模态的保留比例（image_keep_ratio、text_keep_ratio）可实现细粒度控制——例如，保留所有图像/文本 token 以保证质量，同时仍能受益于缓存复用。
• 自适应解掩码：Sprint 不再每一步解掩码固定数量的 token，而是根据模型置信度动态决定揭示多少 token。每一步中，它计算置信度分数（通过 low_confidence、top_k_margin 或 neg_entropy 等策略），并转移置信度最高的 top-k 个 token，其中 k 自适应地设为 ceil(remaining_masked / steps_left)。这样，简单的位置可以快速确定，而算力则集中用于更难的 token。
• 批量接受：在自适应调度的基础上，所有概率超过 threshold 的 token 都会被批量接受，从而进一步减少所需的去噪迭代次数。

图像理解（使用 Sprint）：

response = model.understand_image(
    image_tokens, h, w,
    question="Describe this image in detail.",
    steps=32, gen_length=4096,
    use_sprint=True,
    threshold=0.93,
    keep_ratio=0.5,
    cache_warmup_steps=1,
    image_keep_ratio=1.0,
    text_keep_ratio=1.0,
)

文生图（使用 Sprint）：

result = model.generate_image(
    "A modern Scandinavian kitchen with white cabinetry, marble countertops, and a single orchid on the island. A Nordic woman with sleek blonde ponytail, wearing an oversized sweater and dainty silver necklaces, stirs a matcha bowl with a bamboo whisk, eyes sparkling with quiet joy. Shot with 50mm, f/2.5, diffused window light, cool white balance, low saturation, clean skin retouch. Mood: serene, wholesome, hygge.",
    image_h=1024, image_w=1024,
    cfg_scale=2.0,
    use_sprint=True,
    block_length=32,
    steps=8,
    keep_ratio=0.5,
    cache_warmup_steps=1,
)

[!Note] Sprint 支持 Simple CFG 和无 CFG 模式。当使用 Editing CFG（通过 cfg_text_scale / cfg_image_scale 进行三路引导）时，Sprint 会自动回退到基线模式。

🌟 使用 CLI 脚本

# Text-to-Image
python scripts/t2i_generate.py --model_path inclusionAI/LLaDA2.0-Uni --prompt "A modern Scandinavian kitchen with white cabinetry, marble countertops, and a single orchid on the island. A Nordic woman with sleek blonde ponytail, wearing an oversized sweater and dainty silver necklaces, stirs a matcha bowl with a bamboo whisk, eyes sparkling with quiet joy. Shot with 50mm, f/2.5, diffused window light, cool white balance, low saturation, clean skin retouch. Mood: serene, wholesome, hygge."

# Image Understanding
python scripts/mmu_understand.py --model_path inclusionAI/LLaDA2.0-Uni --image ./assets/understanding_example.png

# Image Editing
python scripts/image_edit.py --model_path inclusionAI/LLaDA2.0-Uni --image ./assets/edit_example.png --instruction "Make it a watercolor painting"

🖥️ ComfyUI 支持

我们提供原生 ComfyUI 自定义节点，用于可视化、基于节点的工作流。全部三项能力（文生图、图像理解、图像编辑）均可作为拖拽式节点使用。

安装

# Symlink into ComfyUI (project must be fully cloned)
cd /path/to/ComfyUI/custom_nodes
ln -s /path/to/LLaDA2.0-Uni/apps/comfyui ./LLaDA2Uni
pip install -r /path/to/LLaDA2.0-Uni/apps/comfyui/requirements.txt

或使用一行安装命令：

bash apps/comfyui/install.sh /path/to/ComfyUI

可用节点

节点	描述
LLaDA2.0_Uni 加载器	使用 Flash Attention / SDPA 加载模型，可选 CPU offload
LLaDA2.0_Uni 文本生成图像	从文本生成图像 token（可选思考模式）
LLaDA2.0_Uni 图像理解	视觉问答
LLaDA2.0_Uni 图像编辑	基于指令的图像编辑
LLaDA2.0_Uni Token 解码器	将 VQ token 解码为像素（turbo：8 步，normal：50 步）
LLaDA2.0_Uni 卸载模型	手动释放 VRAM

工作流示例

# Text-to-Image
Loader → Text-to-Image → Token Decoder → Preview Image

# Image Understanding
Load Image + Loader → Image Understanding → Show Text

# Image Editing
Load Image + Loader → Image Editing → Token Decoder → Preview Image

完整文档请参见 apps/comfyui/README.md。

🚀 SGLang 支持

我们现在支持 SGLang，用于高吞吐量服务和优化推理。

详细配置和示例请参考 cookbook。

⚠️ 许可证

本项目依据 Apache License 2.0 的条款授权。

📖 BibTeX

@article{LLaDA2Uni,
title = {LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model},
author = {Tiwei Bie and Haoxing Chen and Tieyuan Chen and Zhenglin Cheng and Long Cui and Kai Gan and Zhicheng Huang and Zhenzhong Lan and Haoquan Li and Jianguo Li and Tao Lin and Qi Qin and Hongjun Wang and Xiaomei Wang and Haoyuan Wu and Yi Xin and Junbo Zhao},
journal = {arXiv preprint arXiv:2604.20796},
year = {2026}
}