Bringing Block-wise Diffusion to Vision-Language GUI Agents

LLaDA-UI is an MoE-based, block-wise diffusion vision-language GUI agent. It understands screenshots at their native aspect ratio and produces grounded coordinates or structured actions for mobile, desktop, and web interfaces.

This repository contains the model weights. The inference code, example requests, and SGLang serving instructions are available in the LLaDA-UI code repository.

Figure 1. LLaDA-UI GUI-agent performance and qualitative block-wise diffusion decoding. The radar chart and GUI observation remain static while the model output is progressively denoised.

Highlights

• Diffusion-native GUI agent: reasons about screenshots and generates GUI actions with block-wise diffusion decoding rather than autoregressive token-by-token decoding.
• Mixture-of-Experts architecture: approximately 16.7B total parameters.
• Native-resolution vision: preserves the aspect ratio and visual details of mobile, desktop, and web screenshots.
• Cross-platform interaction: supports GUI grounding and agent tasks on mobile, desktop, and web interfaces.
• Structured output: emits normalized coordinates or tagged reasoning and executable actions.

Model Details

Item	Description
Model type	MoE block-wise diffusion vision-language GUI agent
Total parameters	Approximately 16.7B
Language backbone	LLaDA2.0-mini-base
Vision encoder	Native-resolution ViT initialized from SigLIP, with 2D RoPE
Output	Grounded coordinates, text reasoning, and structured GUI actions
Coordinate convention	Integer coordinates normalized to [0, 999]
Supported domains	GUI grounding, mobile, desktop, and web
Weight format	BF16 Safetensors

Inference Pipeline

GUI observations from web, mobile, and desktop environments are encoded at native resolution and combined with task and interaction-history tokens. The LLaDA2.0 decoder progressively denoises the model output into reasoning and executable actions.

Benchmark Results

GUI-agent evaluation reproduced from the technical report. See the code repository for evaluation details and the latest result artifacts.

Quick Start

1. Clone the inference code

git clone https://github.com/inclusionAI/LLaDA-UI.git
cd LLaDA-UI

2. Create an environment

The standalone Hugging Face inference path has been tested with Python 3.10, PyTorch 2.5.1, Transformers 4.51.0, and FlashAttention 2.7.4.post1.

conda create -n llada-ui python=3.10 -y
conda activate llada-ui

pip install torch==2.5.1 torchvision \
  --index-url https://download.pytorch.org/whl/cu124
pip install transformers==4.51.0 Pillow numpy einops accelerate \
  sentencepiece protobuf safetensors
pip install ninja
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir

FlashAttention must be built against a compatible CUDA and PyTorch setup. The checkpoint is about 32 GB in BF16; additional GPU memory is required for model execution and visual tokens.

3. Run single-image grounding

The following command downloads the checkpoint from Hugging Face on first use and returns the center point of the requested UI element:

CUDA_VISIBLE_DEVICES=0 IMAGE_MAX_PIXELS=12845056 \
python -u inference/inference_hf.py \
  --ckpt inclusionAI/LLaDA-UI \
  --image /path/to/screenshot.png \
  --prompt "click the search button" \
  --gen-length 32 \
  --steps 32 \
  --block-length 32

GENERATION: [742,186]
POINT(0~1): [0.742, 0.186]

The raw model coordinate is normalized to [0, 999]. The inference script also prints the corresponding [0, 1] coordinate. To convert it to screen pixels for an image of width W and height H, use (x * W, y * H) with the [0, 1] values.

Large screenshots may also require increasing --max-length beyond its default value of 8192.

SGLang Serving

For agent evaluation or an OpenAI-compatible endpoint, follow the SGLang server guide. The server uses LLaDA-UI-specific multimodal diffusion patches and cannot be started from a stock SGLang installation alone.

After installing the documented serving environment, launch it with:

huggingface-cli download inclusionAI/LLaDA-UI \
  --local-dir /path/to/LLaDA-UI-checkpoint

CUDA_VISIBLE_DEVICES=0,1 SGLANG_DP_SIZE=2 \
bash serve_llada_ui.sh /path/to/LLaDA-UI-checkpoint

Then run one of the packaged requests:

export SGLANG_BASE_URL=http://127.0.0.1:30000/v1
export SGLANG_MODEL=LLaDA-UI

python3 inference/sglang_client.py --example mobile
python3 inference/sglang_client.py --example desktop
python3 inference/sglang_client.py --example web

Output Format

For GUI navigation, a typical response contains tagged reasoning followed by a structured action:

<think>Reasoning grounded in the current GUI state.</think>
<action>Click(box=(x,y))</action>

Here, x and y are integer-normalized to [0, 999]. Despite the field name box, point-based actions contain one interaction coordinate rather than a rectangular bounding box.

For GUI grounding, the model directly returns [x,y]. It returns [-1,-1] when the requested target is infeasible or unrelated to the screenshot.

Citation

@techreport{lladaui2026,
  title  = {LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents},
  author = {Zhangxuan Gu and Haoxing Chen and Qi Qin and Yi Xin and
            Kai Gan and Lin Liu and Long Cui and Xiaomei Wang and
            Beitong Zhou and Yunzhu Zhang and Zhengwen Zeng and
            Changlong Gao and Weizhi Chen and Rongchao Zhang and
            Haoyuan Wu and Shuheng Shen and Changhua Meng and Weiqiang Wang and
            Jianguo Li and Zhenzhong Lan},
  year   = {2026},
  url    = {https://huggingface.co/inclusionAI/LLaDA-UI}
}

17B params

BF16