Google DeepMind 的 Gemma 4 系列多模态模型已在 Hugging Face 上线，支持你常用的智能体、推理引擎和微调库 🤗
这些模型货真价实：以 Apache 2 许可证真正开放，凭借帕累托前沿竞技场分数展现高质量，多模态能力涵盖音频，且尺寸多样，随处可用，包括端侧设备。Gemma 4 建立在先前系列的进展之上，并将它们融会贯通。在我们使用预发布 checkpoint 进行的测试中，其能力令我们印象深刻，以至于我们很难找到好的微调示例，因为它们开箱即用就如此出色。
我们与 Google 及社区合作，让它们随处可用：transformers、llama.cpp、MLX、WebGPU、Rust，应有尽有。这篇博客文章将向你展示如何用你最喜欢的工具进行构建，所以请告诉我们你的想法！
目录
- Gemma 4 有什么新变化？
- Overview of Capabilities and Architecture
- 多模态能力
- Deploy Anywhere
- 多 Token 预测草稿模型
- DiffusionGemma：通过扩散模型进行文本生成
- Fine-tuning & Demos
- 试用 Gemma 4
- 基准测试结果
- 致谢
Gemma 4 有哪些新特性？
与 Gemma-3n 类似，Gemma 4 支持图像、文本和音频输入，并生成文本回复。文本解码器基于 Gemma 模型，支持长上下文窗口。图像编码器与 Gemma 3 的编码器类似，但有两项关键改进：可变宽高比，以及可配置的图像 token 输入数量，以便在速度、内存和质量之间找到最佳平衡点。所有模型都支持图像（或视频）和文本输入，而小型变体（E2B、E4B）和 12B Unified 模型还支持音频。
Gemma 4 提供五种规模，均包含基础版和指令微调版：
| 模型 | 参数量 | 上下文窗口 | 检查点 |
|---|---|---|---|
| Gemma 4 E2B | 有效参数 2.3B，含嵌入向量为 5.1B | 128k | 基础版、指令微调版 |
| Gemma 4 E4B | 有效参数 4.5B，含嵌入向量为 8B | 128k | base、IT |
| Gemma 4 12B Unified | 11.95B 稠密模型，无编码器 | 256K | base、IT |
| Gemma 4 31B | 31B 稠密模型 | 256K | base、IT |
| Gemma 4 26B A4B | 混合专家模型，激活参数 4B / 总参数 26B | 256K | base、IT |
能力与架构概览
Gemma 4 沿用了此前 Gemma 版本及其他开放模型中的若干架构组件，同时舍弃了 Altup 等复杂或结论尚不明确的特性。这一组合是一种经过精心设计的混合方案，旨在实现跨库与跨设备的高度兼容，能够高效支持长上下文和智能体用例，同时非常适合量化。
如上文基准测试所示，这一特性组合（结合训练数据与训练配方）使 31B 稠密模型取得了约 1452 的 LMArena 得分（纯文本），而 26B MoE 仅凭 4B 激活参数就达到了 1441 🤯。正如我们将看到的，多模态表现相对而言与文本生成同样出色，至少在非正式和主观测试中如此。
以下是 Gemma 4 的主要架构特征：
- 交替使用 局部滑动窗口 与 全局全上下文 注意力层。较小的稠密模型使用 512 token 的滑动窗口，而较大的模型使用 1024 token。
- 双 RoPE 配置：滑动层使用标准 RoPE，全局层使用剪枝 RoPE，以支持更长的上下文。
- 逐层嵌入向量（PLE）：第二个嵌入表，向每个解码器层注入一个小的残差信号。
- 共享 KV Cache：模型最后 N 层复用来自更早层的键值状态，从而消除冗余的 KV 投影。
- 视觉编码器（12B 除外）：使用可学习的 2D 位置编码和多维 RoPE。保留原始宽高比，并可将图像编码为几种不同的 token 预算（70、140、280、560、1120）。
- 音频编码器（E2B、E4B）：USM 风格的 conformer，基础架构与 Gemma-3n 中的相同。
- 统一的无编码器多模态（仅 12B）：12B Unified 变体去掉了独立的视觉和音频编码器，将原始图像 patch 和音频波形直接投影到 LLM 的嵌入向量空间。参见 统一多模态（12B）。
逐层嵌入向量（PLE）
较小的 Gemma 4 模型中最具特色的功能之一是逐层嵌入（PLE），这一机制此前已在 Gemma-3n 中引入。在标准 Transformer 中，每个 token 在输入时获得单个嵌入向量，而残差流在所有层中构建所依据的正是这同一个初始表示，这就迫使嵌入必须预先承载模型可能需要的所有信息。
PLE 在主残差流之外增加了一条并行的、维度更低的调制通路。对于每个 token，它通过组合两种信号为每一层生成一个小型专用向量：一个 token 身份分量（来自嵌入查找）和一个上下文感知分量（来自主嵌入的学习投影）。随后，每个解码器层使用其对应的向量，在注意力和前馈网络之后通过一个轻量级残差块来调制隐藏状态。
这使得每一层都有了自己的通道，能够在 token 特定信息变得相关时才接收它，而不需要将所有信息都打包进一个预先的嵌入中。由于 PLE 的维度远小于主隐藏维度，这在适度的参数开销下实现了有意义的逐层特化。对于多模态输入（图像、音频、视频），PLE 在软 token 合并进嵌入序列之前计算——因为 PLE 依赖 token ID，而一旦多模态特征替换了占位符，这些 ID 就会丢失。
多模态位置使用 pad token ID，实际上接收的是中性的逐层信号。
共享 KV Cache
共享 KV cache是一项效率优化，可同时减少推理过程中的计算量和内存占用。模型最后 num_kv_shared_layers 层不计算自己的 key 和 value 投影。相反，它们复用同一注意力类型（滑动或全注意力）的最后一个非共享层的 K 和 V 张量。
在实践中，这对质量的影响微乎其微，同时在长上下文生成和设备端使用方面（无论是内存还是计算）都高效得多。
统一多模态（12B）
Gemma 4 12B 统一模型没有独立的视觉或音频编码器。相反，原始图像块和音频波形通过轻量级线性层直接投影到大语言模型的嵌入向量空间中，所有模态都流入单个仅解码器 Transformer。这降低了多模态延迟，并允许整个模型一次性完成微调。其 checkpoint 大小使其在消费级硬件上部署友好。
多模态能力
在我们的测试中，我们看到 Gemma 4 开箱即支持全面的多模态能力。我们不知道训练数据的具体配比，但我们成功将其用于 OCR、语音转文字、目标检测或指向等任务。它还支持纯文本和多模态的函数调用、推理、代码补全与纠错。
这里，我们展示了跨不同模型规模的几个推理示例。你可以通过 这个 notebook 方便地运行它们。我们鼓励你尝试这些演示，并在本博客下方分享！
多模态输入顺序
Gemma 4 在训练时采用了一种特定的约定来交错输入模态：
- 图像内容放在提示词中文本的前面。
- 音频内容放在提示词中文本的后面。
具体来说，下面是一个为聊天模板准备输入的正确片段：
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
audio_url = "instructions.m4a"
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image_url},
{"type": "text", "text": "Answer in French."},
{"type": "audio", "audio": audio_url},
]
}
]
本文中的音频推理代码片段并未遵循这一约定，因为它们是在该指引得到确认之前编写的。我们保留它们最初编写时的样子，以便结果表格保持可复现。在你自己的代码中，请优先采用上述顺序。
目标检测与指向
GUI 检测
我们在不同尺寸下测试 Gemma 4 的 GUI 元素检测与指向能力，使用以下图像和文本提示词：“图中‘view recipe’元素的边界框是什么？”
使用这个提示词，模型会原生地以 JSON 格式返回检测到的边界框——无需特定指令或语法约束生成。我们发现这些坐标对应 1000x1000 的图像尺寸，相对于输入尺寸而言。
为方便起见，我们在下方将输出可视化。我们从返回的 JSON 中解析边界框：json\n[\n {"box_2d": [171, 75, 245, 308], "label": "view recipe element"}\n]\n
| E2B | E4B |
|---|---|
 |  |
| 26/A4B | 31B |
 |  |
目标检测
我们测试模型检测日常物体的能力，这里我们让它们检测自行车，并比较不同模型的输出。与前一种情况一样，我们从 json 中解析边界框，并转换为图像空间坐标。
| E2B | E4B | 26B/A3B | 31B |
|---|---|---|---|
 |  |  |  |
多模态思考与函数调用
我们让 Gemma 4 编写 HTML 代码，来重建我们用 Gemini 3 制作的一个页面。你可以在下方找到实现这一点的代码，我们启用思考功能，并要求每个模型最多生成 4000 个新 token，以确保万无一失。
| Gemini 生成的网站（参考） | Gemini 复现的图像 |
|---|---|
 |  |
推理代码
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/landing_page.png",
},
{"type": "text", "text": "Write HTML code for this page."},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=4000)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| E2B | E4B |
|---|---|
 |  |
| 31B | 26/A4B |
|---|---|
 |  |
视频理解
较小的 Gemma 4 模型可以接收带音频的视频，而较大的模型可以接收不带音频的视频。虽然这些模型并未在视频上进行显式的后训练，但它们能够理解带音频和不带音频的视频。该模型在音频方面尤为擅长
推理代码
messages = [
{
"role": "user",
"content": [
{"type": "video", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"},
{"type": "text", "text": "What is happening in the video? What is the song about?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
load_audio_from_video=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=200)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
print(result["content"])
| 模型 | 输出 |
|---|---|
| E2B（对音频产生了模型幻觉） | 根据所提供的视频，以下是分析：这个视频里发生了什么？ 视频记录了一场现场音乐演出，很可能是在大型户外音乐节或演唱会上。* 场景： 场景是户外搭建的大型舞台，从庞大的灯光架、大型音箱以及前景中可见的观众可以看出这一点。* 表演者： 舞台上有数位乐手。焦点似乎是一位身穿蓝色衬衫和白色裤子、正在积极表演的主唱兼吉他手，他正在弹奏一把原声吉他或电吉他。背景中还可以看到其他乐手，包括鼓手和贝斯手。* 氛围： 氛围充满活力且极具戏剧性，以明亮的舞台灯光、强烈的蓝色和白色聚光灯以及大量的舞台烟雾为特征，增添了演唱会的气氛。* 观众： 前景中满是观众的后背，表明他们离舞台很近，正投入地观看演出。这首歌是关于什么的？ 视频下方提供的歌词是：> "I don't care 'bout street > From what I cherish > Could it's time to protest |
| E4B ✅ | 视频记录了一支乐队在灯光璀璨、烟雾缭绕的舞台上进行现场演出，周围是观众。乐队为人群表演并演唱。这首歌似乎讲述了现代生活的挣扎与幻灭，尤其是那种被困住的感觉，以及在混乱而令人不堪重负的世界中寻找自身位置的困难。歌词提到了"concrete street"、"broken promises"、"storm of the day"等主题，以及在社会压力中寻找自我的艰难。 |
| 26/A4B（无音频）✅ | 这段视频展示了夜晚一场户外音乐会或音乐节上的现场音乐表演。一位身穿蓝色衬衫和白色长裤的男性独唱歌手是画面的核心焦点。他在一个灯光璀璨的大型舞台上弹奏原声吉他，并对着麦克风演唱。演出由一支乐队伴奏，背景和大型视频屏幕上可以看到鼓手及其他乐手。舞台上弥漫着富有氛围感的蓝色灯光、烟雾和明亮的聚光灯。视频捕捉了不同的视角，包括从观众席仰望舞台的镜头，以及投射在大屏幕上的表演者特写。前景中可以看到正在观看演出的观众。其中一块屏幕上还显示着“Estrella Damm”品牌名称。 |
| 31B（无音频）✅ | 一位音乐人正在演唱会舞台上表演，弹奏原声吉他并对着麦克风演唱。他身穿蓝色衬衫和白色长裤。舞台上弥漫着富有氛围感的烟雾，被明亮的蓝色和白色灯光照亮。舞台上还可以看到其他乐队成员，包括一名鼓手和一名键盘手。视频从观众的视角展示这位音乐人，有些镜头聚焦于他，另一些镜头则展示舞台侧面的大屏幕，屏幕上播放着表演者和观众的特写。 |
我们在图像描述任务上测试了所有模型。所有检查点都表现非常出色，能够准确捕捉复杂场景中的细微之处。以下是我们使用的图像，提示词为“为这张图像写一段详细的描述。”
推理代码
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/bird.png"},
{"type": "text", "text": "Write single detailed caption for this image."},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| 模型 | 输出 |
|---|---|
| E2B ✅ | 一个中景镜头捕捉到一只饱经风霜的海鸥栖息在石质基座顶端，场景似乎是一个熙熙攘攘的欧洲广场，画面右侧占据主导地位的是一座宏伟的古典风格建筑，带有华丽的立柱和建筑细节。背景中，一栋低矮的浅色建筑横贯中景，两侧点缀着一些绿色植被，还能隐约看到其他建筑，包括一栋赤陶屋顶的建筑。前景中，可以看到一个现代公交站或候车亭的一部分，带有指示路线的标牌，整个场景沐浴在明亮的天光下，天空呈淡白色。 |
| E4B ✅ | 一只孤零零的海鸥栖息在熙熙攘攘的欧洲广场上一座灰色石质基座顶端，画面右侧占据主导地位的是一座古典建筑的历史宏伟感，与背景中更为现代的低层建筑形成对比，天空明亮而朦胧。 |
| 26B/A4B ✅ | 一个全景镜头展示了一只白棕相间的海鸥栖息在意大利威尼斯的一根灰色柱子上。海鸥面朝左侧，喙指向画面边缘。它身后是一栋浅灰色两层建筑，带有几扇深色矩形窗户。海鸥右侧是一栋大型白色建筑，带有立柱和精美的雕刻。天空是清澈的浅蓝色。光线明亮而均匀，表明是白天。 |
| 31B ✅ | 一个高角度中景镜头显示，一只白灰相间的海鸥栖息在意大利威尼斯一根灰色混凝土塔柱上。海鸥面朝左侧，身体略微朝向镜头。它白色的头部、颈部和胸部与灰色的翅膀和背部形成对比。海鸥的橙色鸟喙微微张开。塔柱是一根浅灰色混凝土圆柱，表面带有风化的纹理。背景中有几栋建筑。左侧是一栋长而低矮的白色建筑，带有一排小型矩形窗户。中央是一簇绿树。右侧是一栋红色建筑，带有瓦片屋顶和几扇小型矩形窗户。再往右是一栋大型、华丽的白石建筑，带有若干立柱和雕塑。天空呈淡蓝色，光线明亮而清澈。左下角有一辆黄色公交车，车身上用黑色字母印着“Ferrovia”字样。背景中有几个人在行走，但他们很小且处于失焦状态。 |
音频问答
这些模型经过训练，用于回答关于音频中语音的问题。音乐和非语音声音不属于训练数据的一部分。
推理代码
messages = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"},
{"type": "text", "text": "Can you describe this audio in detail?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(
**inputs,
max_new_tokens=1000,
do_sample=False,
)
print(processor.decode(output[0], skip_special_tokens=True))
| 模型 | 输出 |
|---|---|
| E2B | 这段音频是一段个人反思。讲述者正在谈论他们向全国发表的告别演说，这场演说是在芝加哥发表的。他们表达了对与美国民众所进行对话的感激之情，并指出尽管未曾与他们面对面相见、甚至未曾打过招呼，但这些在客厅、学校、农场、工厂车间、餐馆和军事哨所等不同场合的互动，正是支撑他们一路走来的动力。 |
| E4B | 这段音频是一段演讲节选，讲述者正在芝加哥向全国发表告别演说。讲述者回顾了自己的执政时光，对与美国民众在客厅、学校、农场、工厂、餐馆和军事哨所等不同场合所进行的对话表达了感激之情。语气充满反思与感恩，凸显了这些互动在其政治历程中的重要意义。 |
如果你想进行转录，这里有一个示例：
推理代码
messages = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"},
{"type": "text", "text": "Transcribe the audio?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(
**inputs,
max_new_tokens=1000,
do_sample=False,
)
print(processor.decode(output[0], skip_special_tokens=True))
| 模型 | 输出 |
|---|---|
| E2B | 本周，我前往芝加哥，遵循历任总统的传统，向全国发表我的最后告别演说。这是一个表达感谢的机会。无论我们是否意见一致，还是几乎从未达成共识，我与你们——美国人民——在客厅和学校里、在农场和工厂车间里、在餐馆和遥远的军事前哨的每一次对话，所有这些对话正是让我保持诚实的原因。 |
| E4B | 本周，我前往芝加哥，遵循历任总统的传统，向全国发表我的最后告别演说。这是一个表达感谢的机会。无论我们是否意见一致，还是几乎从未达成共识，我与你们——美国人民——在客厅和学校里、在农场和工厂车间里、在餐馆和遥远的军事前哨的每一次对话，所有这些对话正是让我保持诚实的原因。 |
多模态函数调用
我们通过要求获取图片所示地点的天气来测试该模型。
推理代码
import re
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_weather",
"description": "Gets the current weather for a specific location.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "The city name"},
},
"required": ["city"],
},
},
}
tools = [WEATHER_TOOL]
messages = [
{"role": "user", "content": [
{"type": "text", "text": "What is the city in this image? Check the weather there right now."},
{"type": "image", "image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/thailand.jpg"},
]},
]
inputs = processor.apply_chat_template(
messages,
tools=[WEATHER_TOOL],
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=1000)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| 模型 | 输出 |
|---|---|
| E2B | 分析请求： 用户提供了一张图片并提出了两个问题：\n * “这张图片中是哪座城市？” * “查一下那里现在的天气。”\n\n2. 分析图片： 图片展示了一座大型、华丽、高耸的建筑，具有寺庙建筑的特征，很可能位于东南亚（从风格判断）。它似乎是一处著名地标。 3. 识别地标/位置（内部知识/视觉线索）： 该建筑与泰国曼谷等地常见的佛塔/窣堵坡（如 Wat Arun 或类似建筑）高度相似。\n\n4. 确定城市： 根据视觉证据，该地点极有可能是泰国曼谷。\n\n5. 检查可用工具： 可用的工具是 get_weather(city: STRING)。 6. **制定计划：**\n * 第 1 步：从图片中识别城市（曼谷）。\n * 第 2 步：使用 get_weather 工具，传入识别出的城市。\n\n7. 构造工具调用： get_weather(city="Bangkok")\n\n8. 生成回复： 模型应首先说明识别出的城市，然后调用工具获取天气。（自我纠正：如果模型需要外部信息，它必须以正确的格式调用工具。）call:get_weather{city:Bangkok} |
| E4B | '1. 分析请求： 用户提供了一张图片，并提出了两件事：\n a. 识别图片中的城市。\n b. 查询该城市当前的天气。\n\n2. 分析图片： 图片展示了一座非常独特、装饰华丽的塔/尖顶结构，具有泰国建筑的典型特征（具体来说，是一座 chedi 或 prang）。\n\n3. 确定城市识别（内部知识/视觉识别）： 基于建筑风格，该结构非常类似于泰国的著名寺庙，例如曼谷的 Wat Arun。\n\n4. 评估工具可用性： 我有一个可用工具：get_weather(city: STRING)。该工具需要城市名称作为输入。\n\n5. **制定计划：**\n a. 我需要城市名称才能使用 get_weather 工具。\n b. 我必须先从图片中识别出城市。\n c. 由于我的内部知识表明是曼谷，我将基于这一假设继续，但我必须承认，仅凭图片我无法确定性地证明其位置。\n\n6. 执行第 1 步（城市识别）： 基于建筑的视觉证据，假设城市为曼谷。\n\n7. 执行第 2 步（生成工具调用）： 使用 get_weather 工具，并以 "Bangkok" 作为 city 参数。\n\n8. 构建响应： 以 JSON 格式生成相应的工具调用。', 'answer': '<|tool_call>call:get_weather{city:<|"|>Bangkok<|"|>}<tool_call|>' |
| 31B | 用户想知道图片中的城市以及当地当前的天气。1. 分析图片： 图片中展示了装饰华丽的尖塔状结构（prangs），这是泰国寺庙建筑的典型特征。具体来说，这些看起来像是泰国曼谷玉佛寺（Wat Phra Kaew）或卧佛寺（Wat Pho）的 prangs。这种风格是曼谷皇家寺庙建筑群非常鲜明的特征。2. 识别城市： 根据建筑风格，这座城市是曼谷。3. 查询天气： 我需要调用 get_weather 工具，使用 city="Bangkok".call:get_weather{city:Bangkok} |
| 26/A4B | 图片中展示了几座高耸、装饰华丽的塔（prangs），这是泰国佛教寺庙建筑的典型特征。具体来说，这些看起来像是泰国曼谷大皇宫或玉佛寺内的建筑。这种风格无疑是泰式的。我会通过查看具体的塔来确认这一点。这些非常具有标志性。\n2. 确认地点： 图片中展示了曼谷大皇宫建筑群的元素。\n3. 获取天气信息： 一旦确定城市为曼谷，我将使用 get_weather 工具，使用 city='Bangkok'.call:get_weather{city:Bangkok} |
随处部署
Gemma 4 从发布首日就支持众多开源推理引擎，非常适合工具调用和智能体！我们还发布了可在多种硬件后端上运行的 ONNX checkpoint，从而支持边缘设备或浏览器中的使用场景！
transformers
Gemma 4 从一开始就自带一流的 transformers 支持 🤗。这一集成让你可以将该模型与 bitsandbytes、PEFT 和 TRL 等其他库配合使用。请务必安装最新版本的 transformers。
pip install -U transformers
使用小型 Gemma 4 模型进行推理最简单的方式是通过 any-to-any pipeline。你可以按如下方式初始化它。
from transformers import pipeline
pipe = pipeline("any-to-any", model="google/gemma-4-e2b-it")
然后你可以按如下方式传入图像和文本。
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/thailand.jpg",
},
{"type": "text", "text": "Do you have travel advice going to here?"},
],
}
]
output = pipe(messages, max_new_tokens=100, return_full_text=False)
output[0]["generated_text"]
在使用视频进行推理时，你可以通过 load_audio_from_video 参数加入音轨。
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/rockets.mp4",
},
{"type": "text", "text": "What is happening in this video?"},
],
}
]
pipe(messages, load_audio_from_video=True)
再往底层走一层，你可以使用 AutoModelForMultimodalLM 类加载 Gemma 4，这在微调时尤其有用。内置的聊天模板会负责正确地格式化输入，请务必使用它，以免在手动构建提示词时出现细微错误。
推理代码
from transformers import AutoModelForMultimodalLM, AutoProcessor
model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-E2B-it", device_map="auto")
processor = AutoProcessor.from_pretrained("google/gemma-4-E2B-it")
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/rockets.mp4",
},
{"type": "text", "text": "What is happening in this video?"},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Llama.cpp
Gemma 4 模型从一开始就在 llama.cpp 中支持图像+文本！这让你可以在所有你喜爱的本地应用中使用 Gemma 4：llama-cpp server、lmstudio、Jan，以及像 Pi 这样的编码智能体，并跨 Metal 和 CUDA 等多种后端运行。
你可以按如下方式安装 llama-cpp。
curl -LsSf https://llama.app/install.sh | sh
然后你可以启动一个与 OpenAI API 兼容的服务器。将命令末尾的量化方案替换为你想要的精度。
llama serve -hf ggml-org/gemma-4-E2B-it-GGUF
查看此链接 了解更多关于将 llama.cpp 与不同编码智能体和本地应用结合的选项。在此集合中找到所有 GGUF 检查点。
接入你的本地智能体
我们致力于确保新模型能在本地与 openclaw、hermes、pi 和 open code 等智能体配合运行。这一切都归功于 llama.cpp！运行以下命令即可立即试用 Gemma 4。
首先，启动你的本地服务器：
llama serve -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M
对于 hermes：
hermes model
对于 openclaw：
openclaw onboard
对于 pi，定义一个 ~/.pi/agent/models.json：
{
"providers": {
"llama-cpp": {
"baseUrl": "http://localhost:8080/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [
{
"id": "ggml-org-gemma-4-26b-4b-gguf"
}
]
}
}
}
对于 open code，定义一个 ~/.config/opencode/opencode.json：
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama.cpp": {
"npm": "@ai-sdk/openai-compatible",
"name": "llama server (local)",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"gemma-4-26b-4b-it": {
"name": "Gemma 4 (local)",
"limit": {
"context": 128000,
"output": 8192
}
}
}
}
}
}
transformers.js
transformers.js 支持直接在浏览器中运行 Gemma 4。你可以查看模型卡片，在此处详细了解纯文本、图像与文本、音频与文本推理。我们还发布了一个演示，供你在此处测试该模型。
MLX
Gemma 4 的完整多模态支持可通过开源 mlx-vlm 库 使用。以下是如何让模型描述一张图像：
pip install -U mlx-vlm
mlx_vlm.generate \
--model google/gemma-4-E4B-it \
--image https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg \
--prompt "Describe this image in detail"
mlx-vlm 支持 TurboQuant，它在保持与未压缩基线相同精度的同时，仅使用约 4 倍的活跃内存，并且端到端运行速度快得多。这使得在 Apple Silicon 上进行长上下文推理变得切实可行，而无需牺牲质量。使用方法如下：
mlx_vlm.generate \
--model "mlx-community/gemma-4-26b-a4b-it-4bit" \
--prompt "Your prompt here" \
--kv-bits 3.5 \
--kv-quant-scheme turboquant
如需音频示例和更多细节，请查看 MLX 合集。
Mistral.rs
mistral.rs 是一个 Rust 原生的推理引擎，在发布首日即支持 Gemma 4 的所有模态（文本、图像、视频、音频），并内置工具调用和智能体功能。安装 mistral.rs：
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh
irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
然后你可以启动一个兼容 OpenAI 的 HTTP 服务器：
mistralrs serve mistralrs-community/gemma-4-E4B-it-UQFF --from-uqff 8
或者，使用交互模式：
mistralrs run -m google/gemma-4-E4B-it --isq 8 --image image.png -i "Describe this image in detail."
mistralrs run -m google/gemma-4-E4B-it --isq 8 --audio audio.mp3 -i "Transcribe this fully."
在 此处 查找所有模型。请遵循模型卡片中的 说明 进行安装和推理。
多 Token 预测草稿模型
Google 为 Gemma 4 系列发布了多 Token 预测（MTP）草稿模型：这些小型助手模型通过投机解码加速推理。草稿模型一次性提出多个未来 Token，目标模型则在单次前向传播中对其进行验证。你得到的输出与目标模型完全一致，只是速度更快——没有质量损失，也不改变推理行为。据报告，端到端加速最高可达约 3 倍，具体取决于硬件、批大小和工作负载。
助手模型覆盖 Gemma 4 的全部四种规模（E2B、E4B、26B A4B、31B）。它们与目标模型共享 KV cache，以避免重新计算上下文，而更小的边缘变体还额外采用嵌入向量聚类技巧，以在设备端保持低内存和低计算量。
你可以在 Gemma 4 合集和 mlx-community 合集中找到这些检查点。
DiffusionGemma：通过扩散模型进行文本生成
在自回归的 Gemma 4 系列之外，Google DeepMind 还发布了 DiffusionGemma，这是一个多模态模型，使用离散扩散而非逐 Token 自回归来生成文本。它构建在相同的 26B A4B 混合专家（MoE）基础之上（总参数 25.2B / 激活参数 3.8B，128 个专家中激活 8 个，外加 1 个共享专家），接受文本和图像输入，生成文本，并支持最高 256K 上下文——全部采用相同的 Apache 2.0 许可证。
标准的因果语言模型一次只生成一个 token，而 DiffusionGemma 则并行地对整块 token 进行去噪。其架构为编码器-解码器：自回归编码器对提示词进行预填充并构建 KV cache，而解码器则在一个 256 token 的“画布”上应用双向注意力。在多画布采样过程中，模型使用扩散采样器对整块画布进行迭代去噪；一旦某块画布定稿，它就会被编码并追加到 KV cache 中，随后开始下一块画布。这种块自回归方法提升了生成速度。
最突出的优势是吞吐量：并行去噪每次前向传播大约生成 15–20 个 token，在低批量规模下（H100，FP8）可达到超过 1100 token/秒的单用户生成速度。推理算力也是自适应的——更简单的提示词以及代码等结构化任务需要更少的去噪步骤，因此每秒 token 数会随任务复杂度而变化。它保留了更广泛的 Gemma 4 工具集：思考模式、函数调用、长上下文、原生系统提示词，以及在可变宽高比和分辨率下的图像理解（OCR、文档解析、目标检测、指向）。
基准测试显示出速度与评估指标之间预期中的权衡——DiffusionGemma 在大多数任务上落后于自回归的 26B A4B（例如 MMLU Pro 77.6% 对 82.6%，AIME 2026 69.1% 对 88.3%，GPQA Diamond 73.2% 对 82.3%），以换取其巨大的速度优势，同时在少数任务上略占上风（HLE 无工具 11.0% 对 8.7%）。
上手方式看起来与 Gemma 4 的其余部分完全一样，通过专用的 diffusion 类：
from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor
MODEL_ID = "google/diffusiongemma-26B-A4B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
MODEL_ID,
dtype="auto",
device_map="auto",
)
message = [{"role": "user", "content": "Why is the sky blue?"}]
input_ids = processor.apply_chat_template(
message,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**input_ids, max_new_tokens=512)
text = processor.decode(output[0], skip_special_tokens=False)
为获得最佳效果，Google 推荐使用 熵约束（EB）采样器，并配合自适应停止（最多 48 个去噪步骤，温度调度从 0.8 → 0.4 衰减，token 选择的熵约束为 0.1）。与 Gemma 4 一样，请将图像内容放在提示词中文本的前面，并通过 <|think|> 控制 token 来切换推理。
人人皆可微调
Gemma 4 模型非常适合在你喜爱的工具和平台上进行微调，且不受预算限制。
使用 TRL 进行微调
Gemma 4 已全面支持使用 TRL 进行微调。为庆祝这一进展，TRL 已升级，支持在与环境交互时的多模态工具响应，这意味着模型现在可以在训练期间从工具接收图像，而不仅仅是文本。
为展示这一能力，我们构建了一个示例训练脚本，让 Gemma 4 在 CARLA 模拟器中学习驾驶。模型通过摄像头观察道路，决定该做什么，并从结果中学习。训练后，它能持续变道以避开行人。同样的方法适用于任何需要模型观察并行动的任务：机器人、网页浏览或其他交互式环境。
开始使用：
# pip install git+https://github.com/huggingface/trl.git
python examples/grpo_carla/carla_vlm_gemma.py \
--env-urls https://sergiopaniego-carla-env.hf.space \
https://sergiopaniego-carla-env-2.hf.space \
--model google/gemma-4-E2B-it
在此处查看示例。
在 Vertex AI 上使用 TRL 进行微调
此外，我们还准备了一个示例，展示如何在 Vertex AI 上使用 TRL 通过 SFT 对 Gemma 4 进行微调，以演示如何在冻结视觉和音频塔的同时扩展函数调用能力。这些示例包括如何在 Google Cloud 上构建带有 CUDA 支持、包含最新 Transformers、TRL 等的自定义 Docker 容器，以及如何通过 Vertex AI Serverless Training Jobs 运行它。
from google.cloud import aiplatform
aiplatform.init(
project="<PROJECT_ID>",
location="<LOCATION>",
staging_bucket="<BUCKET_URI>",
)
job = aiplatform.CustomContainerTrainingJob(
display_name="gemma-4-fine-tuning",
container_uri="<CONTAINER_URI>",
command=["python", "/gcs/gemma-4-fine-tuning/train.py"],
)
job = job.submit(
replica_count=1,
machine_type="a3-highgpu-1g",
accelerator_type="NVIDIA_H100_80GB",
accelerator_count=1,
base_output_dir="<BUCKET_URI>/output-dir",
environment_variables={
"MODEL_ID": "google/gemma-4-E2B-it",
"HF_TOKEN": <HF_TOKEN>,
},
boot_disk_size_gb=500,
)
你可以在“Hugging Face on Google Cloud”文档中找到完整示例：https://hf.co/docs/google-cloud/examples/vertex-ai-notebooks-fine-tune-gemma-4。
使用 Unsloth Studio 进行微调
如果你想在 UI 中微调并运行 Gemma 4 模型，可以试试 Unsloth Studio。它可以在本地或 Google Colab 上运行。首先，安装并启动该应用：
# install unsloth studio on MacOS, Linux, WSL
curl -fsSL https://unsloth.ai/install.sh | sh
# install unsloth studio on Windows
irm https://unsloth.ai/install.ps1 | iex
# launch unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Search for for a Gemma 4 model like google/gemma-4-E2B-it
然后从 hub 中选择任意一个 Gemma 4 模型。
试用 Gemma 4
我们已发布了一些演示，供你试用不同的 Gemma 4 模型。我们提供了基于 transformers 实现的 E4B、12B Unified、26B/A4B MoE 和 31B dense 模型的演示。还有一个使用 transformers.js 的 WebGPU 演示 🚀
基准测试结果
Gemma 4 模型在从推理、编程到视觉和长上下文任务等各类基准测试中均展现出卓越性能。下图展示了模型性能与规模的关系，Gemma 4 模型构成了一条令人瞩目的帕累托前沿：
来源：Google（blog.google）
以下是指令微调模型的详细基准测试结果：
| 基准测试 | Gemma 4 31B | Gemma 4 26B A4B | Gemma 4 12B Unified | Gemma 4 E4B | Gemma 4 E2B | Gemma 3 27B（无思考） |
|---|---|---|---|---|---|---|
| 推理与知识 | ||||||
| MMLU Pro | 85.2% | 82.6% | 77.2% | 69.4% | 60.0% | 67.6% |
| AIME 2026 无工具 | 89.2% | 88.3% | 77.5% | 42.5% | 37.5% | 20.8% |
| GPQA Diamond | 84.3% | 82.3% | 78.8% | 58.6% | 43.4% | 42.4% |
| Tau2（3 次平均） | 76.9% | 68.2% | 69.0% | 42.2% | 24.5% | 16.2% |
| BigBench Extra Hard | 74.4% | 64.8% | 53.0% | 33.1% | 21.9% | 19.3% |
| MMMLU | 88.4% | 86.3% | 83.4% | 76.6% | 67.4% | 70.7% |
| 编程 | ||||||
| LiveCodeBench v6 | 80.0% | 77.1% | 72.0% | 52.0% | 44.0% | 29.1% |
| Codeforces ELO | 2150 | 1718 | 1659 | 940 | 633 | 110 |
| HLE 无工具 | 19.5% | 8.7% | 5.2% | - | - | - |
| 带搜索的 HLE | 26.5% | 17.2% | - | - | - | - |
| 视觉 | ||||||
| MMMU Pro | 76.9% | 73.8% | 69.1% | 52.6% | 44.2% | 49.7% |
| OmniDocBench 1.5（编辑距离） | 0.131 | 0.149 | 0.164 | 0.181 | 0.290 | 0.365 |
| MATH-Vision | 85.6% | 82.4% | 79.7% | 59.5% | 52.4% | 46.0% |
| MedXPertQA MM | 61.3% | 58.1% | 48.7% | 28.7% | 23.5% | - |
| 音频 | ||||||
| CoVoST | - | - | 38.5* | 35.54 | 33.47 | - |
| FLEURS（越低越好） | - | - | 0.069* | 0.08 | 0.09 | - |
| 长上下文 | ||||||
| MRCR v2 8 needle 128k（平均） | 66.4% | 44.1% | 43.4% | 25.4% | 19.1% | 13.5% |
*不含中文。
致谢
这项工作离不开 Google 在模型产物上的大量贡献，也离不开其为将模型贡献到 transformers 以实现标准化所付出的巨大努力。如今开源生态更加完整，拥有了一个能力出众、可自由许可的开源模型。Gemma 4 的 transformers 集成由 Cyril、Raushan、Eustache、Arthur、Lysandre 负责。
我们感谢 Joshua 完成 transformers.js 的集成与演示，Eric 完成 mistral.rs 集成，Son 完成 Llama.cpp，Prince 完成 MLX，Quentin、Albert 和 Kashif 完成 TRL，Adarsh 完成 SGLang transformers 后端，以及 Toshihiro 构建了多个演示。
The Gemma 4 family of multimodal models by Google DeepMind is out on Hugging Face, with support for your favorite agents, inference engines, and fine-tuning libraries 🤗
These models are the real deal: truly open with Apache 2 licenses, high quality with pareto frontier arena scores, multimodal including audio, and sizes you can use everywhere including on-device. Gemma 4 builds on advances from previous families and makes them click together. In our tests with pre-release checkpoints we have been impressed by their capabilities, to the extent that we struggled to find good fine-tuning examples because they are so good out of the box.
We collaborated with Google and the community to make them available everywhere: transformers, llama.cpp, MLX, WebGPU, Rust; you name it. This blog post will show you how to build with your favorite tools so let us know what you think!
Table of Contents
- What is New with Gemma 4?
- Overview of Capabilities and Architecture
- Multimodal Capabilities
- Deploy Anywhere
- Multi-Token Prediction Drafters
- DiffusionGemma: Text Generation via Diffusion
- Fine-tuning & Demos
- Try Gemma 4
- Benchmark Results
- Acknowledgements
What is new with Gemma 4?
Similar to Gemma-3n, Gemma 4 supports image, text, and audio inputs, and generates text responses. The text decoder is based on the Gemma model with support for long context windows. The image encoder is similar to the one from Gemma 3 but with two crucial improvements: variable aspect ratios, and configurable number of image token inputs to find your sweet spot between speed, memory, and quality. All models support images (or video) and text inputs, while the small variants (E2B, E4B) and the 12B Unified model support audio as well.
Gemma 4 comes in five sizes, all base and instruction fine-tuned:
| Model | Parameter Size | Context Window | Checkpoints |
|---|---|---|---|
| Gemma 4 E2B | 2.3B effective, 5.1B with embeddings | 128k | base, IT |
| Gemma 4 E4B | 4.5B effective, 8B with embeddings | 128k | base, IT |
| Gemma 4 12B Unified | 11.95B dense, encoder-free | 256K | base, IT |
| Gemma 4 31B | 31B dense model | 256K | base, IT |
| Gemma 4 26B A4B | mixture-of-experts with 4B activated/26B total parameters | 256K | base, IT |
Overview of Capabilities and Architecture
Gemma 4 leverages several architecture components used in previous Gemma versions and other open models, and leaves out complex or inconclusive features such as Altup. The combination is a mix designed to be highly compatible across libraries and devices, that can efficiently support long context and agentic use cases, whilst being ideal for quantization.
As shown in the benchmarks above, this feature mix (combined with the training data and recipe) enables the 31B dense model to achieve an estimated LMArena score (text only) of 1452, while the 26B MoE reaches 1441 with just 4B active parameters 🤯. As we'll see, multimodal operation is comparatively as good as text generation, at least in informal and subjective tests.
These are the main architecture characteristics in Gemma 4:
- Alternating local sliding-window and global full-context attention layers. Smaller dense models use sliding windows of 512 tokens while larger models use 1024 tokens.
- Dual RoPE configurations: standard RoPE for sliding layers, pruned RoPE for global layers, to enable longer context.
- Per-Layer Embeddings (PLE): a second embedding table that feeds a small residual signal into every decoder layer.
- Shared KV Cache: the last N layers of the model reuse key-value states from earlier layers, eliminating redundant KV projections.
- Vision encoder (except 12B): uses learned 2D positions and multidimensional RoPE. Preserves the original aspect ratios and can encode images to a few different token budgets (70, 140, 280, 560, 1120).
- Audio encoder (E2B, E4B): USM-style conformer with the same base architecture as the one in Gemma-3n.
- Unified encoder-free multimodality (12B only): the 12B Unified variant drops the separate vision and audio encoders and projects raw image patches and audio waveforms directly into the LLM's embedding space. See Unified Multimodal (12B).
Per-Layer Embeddings (PLE)
One of the most distinctive features in smaller Gemma 4 models is Per-Layer Embeddings (PLE), which was introduced previously in Gemma-3n. In a standard transformer, each token gets a single embedding vector at input, and the same initial representation is what the residual stream builds on across all layers, forcing the embedding to frontload everything the model might need. PLE adds a parallel, lower-dimensional conditioning pathway alongside the main residual stream. For each token, it produces a small dedicated vector for every layer by combining two signals: a token-identity component (from an embedding lookup) and a context-aware component (from a learned projection of the main embeddings).
Each decoder layer then uses its corresponding vector to modulate the hidden states via a lightweight residual block after attention and feed-forward. This gives each layer its own channel to receive token-specific information only when it becomes relevant, rather than requiring everything to be packed into a single upfront embedding. Because the PLE dimension is much smaller than the main hidden size, this adds meaningful per-layer specialization at modest parameter cost. For multimodal inputs (images, audio, video), PLE is computed before soft tokens are merged into the embedding sequence — since PLE relies on token IDs that are lost once multimodal features replace the placeholders.
Multimodal positions use the pad token ID, effectively receiving neutral per-layer signals.
Shared KV Cache
The shared KV cache is an efficiency optimization that reduces both compute and memory during inference. The last num_kv_shared_layers layers of the model don't compute their own key and value projections. Instead, they reuse the K and V tensors from the last non-shared layer of the same attention type (sliding or full).
In practice, this has a minimal impact on quality while being much more efficient (in terms of both memory and compute) for long context generation and on-device use.
Unified Multimodal (12B)
The Gemma 4 12B Unified model has no separate vision or audio encoder. Instead, raw image patches and audio waveforms are projected directly into the LLM's embedding space through lightweight linear layers, and all modalities flow into a single decoder-only transformer. This reduces multimodal latency and allows the whole model to be fine-tuned in one pass. The checkpoint size makes it deployment-friendly on consumer hardware.
Multimodal Capabilities
We saw in our tests that Gemma 4 supports comprehensive multimodal capabilities out of the box. We don't know what was the training mix, but we had success using it for tasks such as OCR, speech-to-text, object detection, or pointing. It also supports text-only and multimodal function calling, reasoning, code completion and correction.
Here, we show a few inference examples across different model sizes. You can run them conveniently with this notebook. We encourage you to try the demos and share them below this blog!
Multimodal Input Order
Gemma 4 was trained with a specific convention to interleave input modalities:
- Image content goes before the text in your prompt.
- Audio content goes after the text in your prompt.
To be specific, this is a correct fragment that prepares inputs for the chat template:
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
audio_url = "instructions.m4a"
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image_url},
{"type": "text", "text": "Answer in French."},
{"type": "audio", "audio": audio_url},
]
}
]
The audio inference snippets in this post don't follow this convention, as they were written before this guidance was confirmed. We are keeping them as originally written so result tables remain reproducible. For your own code, prefer the order above.
Object Detection and Pointing
GUI detection
We test Gemma 4 on GUI element detection and pointing across different sizes, with the following image and text prompt: "What's the bounding box for the "view recipe" element in the image?"
With this prompt, the model natively responds in JSON format with the detected bounding boxes - no need for specific instructions or grammar-constrained generation. We found the coordinates refer to an image size of 1000x1000, relative to the input dimensions.
We visualize the outputs below for your convenience. We parse the bounding boxes from the returned JSON: json\n[\n {"box_2d": [171, 75, 245, 308], "label": "view recipe element"}\n]\n
| E2B | E4B |
|---|---|
| |
| 26/A4B | 31B |
| |
Object Detection
We test models to detect everyday objects, here we ask them to detect the bike and compare different model outputs. As in the previous case, we parse the bounding box from the json and translate to image space coordinates.
| E2B | E4B | 26B/A3B | 31B |
|---|---|---|---|
| | | |
Multimodal Thinking and Function Calling
We asked Gemma 4 to write HTML code to reconstruct a page we made with Gemini 3. Below you can find the code to do this, we enable thinking and ask each model to generate up to 4000 new tokens, to make it foolproof.
| Gemini Generated Website (Reference) | Gemini Reproduced Image |
|---|---|
| |
Inference code
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/landing_page.png",
},
{"type": "text", "text": "Write HTML code for this page."},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=4000)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| E2B | E4B |
|---|---|
| |
| 31B | 26/A4B |
|---|---|
| |
Video Understanding
Smaller Gemma 4 models can take in videos with audio while larger ones can take in videos without audio. While the models are not explicitly post-trained on videos, they can understand videos both with and without audio. The model is particularly strong in audios
Inference code
messages = [
{
"role": "user",
"content": [
{"type": "video", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"},
{"type": "text", "text": "What is happening in the video? What is the song about?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
load_audio_from_video=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=200)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
print(result["content"])
| Model | Output |
|---|---|
| E2B (hallucinated for the audio) | Based on the video provided, here is an analysis: What is happening in this video? The video captures a live musical performance, likely at a large outdoor festival or concert. * Setting: The scene is a large stage set up outdoors, indicated by the extensive lighting rig, large speakers, and the audience visible in the foreground. * Performers: There are several musicians on stage. The focal point seems to be a lead singer/guitarist in a blue shirt and white pants who is actively performing, playing an acoustic or electric guitar. There are also other musicians, including drummers and bassists, visible in the background. * Atmosphere: The atmosphere is energetic and dramatic, characterized by bright stage lights, intense blue and white spotlights, and a significant amount of stage fog/smoke, which adds to the concert mood. * Audience: The foreground is filled with the backs of audience members, showing they are close to the stage and engaged in watching the performance. What is the song about? The lyrics provided below the video are: > "I don't care 'bout street > From what I cherish > Could it's time to protest |
| E4B ✅ | The video captures a live performance of a band on a stage with bright lights and smoke effects, surrounded by an audience. The band performs and sings to the crowd. The song appears to be about the struggles and disillusionment of modern life, specifically the feeling of being stuck and the difficulties in finding one's place in a chaotic and overwhelming world. Lyrics mention themes of the "concrete street," "broken promises," "storm of the day," and the difficulty of finding oneself amidst societal pressures. |
| 26/A4B (no audio) ✅ | This video shows a live music performance at an outdoor concert or festival at night. A solo male artist, dressed in a blue shirt and white pants, is the central focus. He is playing an acoustic guitar and singing into a microphone on a large, lit stage. The performance is accompanied by a band, including a drummer and other musicians visible in the background and on large video screens. The stage is filled with atmospheric blue lighting, smoke, and bright spotlights. The video captures different perspectives, including shots from the crowd looking up at the stage and close-up views of the performers projected onto large screens. The crowd is visible in the foreground, watching the show. One of the screens also displays the "Estrella Damm" brand name. |
| 31B (no audio) ✅ | A musician is performing on stage during a concert, playing an acoustic guitar and singing into a microphone. He is wearing a blue shirt and white pants. The stage is filled with atmospheric smoke and illuminated by bright blue and white lights. Other band members, including a drummer and a keyboardist, are also visible on stage. The video shows the musician from the perspective of the audience, with some shots focusing on him and others showing the large screen on the side of the stage, which displays close-ups of the performer and the crowd. |
We have tested all models on captioning. All checkpoints perform very well and accurately capture nuance in complex scenerios. Here's the image we use, with the prompt "Write single detailed caption for this image.".
Inference code
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/bird.png"},
{"type": "text", "text": "Write single detailed caption for this image."},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| Model | Output |
|---|---|
| E2B ✅ | A medium shot captures a weathered seagull perched atop a stone pedestal in what appears to be a bustling European square, with a grand, classical-style building featuring ornate columns and architectural details dominating the right side of the frame. In the background, a low, light-colored building stretches across the middle ground, flanked by some green foliage, and hints of other structures, including a terracotta-roofed building, are visible. In the foreground, part of a modern bus stop or shelter is visible, with signage indicating routes, and the overall scene is bathed in bright daylight under a pale sky. |
| E4B ✅ | A solitary seagull perches atop a grey stone pedestal in a bustling European piazza, with the historical grandeur of a classical building dominating the right side of the frame, contrasting with the more modern, low-rise architecture in the background under a bright, hazy sky. |
| 26B/A4B ✅ | A full shot shows a white and brown seagull perched on a gray pillar in Venice, Italy. The seagull faces left, its beak pointed towards the edge of the frame. Behind it is a light gray, two-story building with several dark rectangular windows. To the right of the seagull is a large, white building with columns and intricate carvings. The sky is a clear, light blue. The lighting is bright and even, suggesting it is daytime. |
| 31B ✅ | A high-angle, medium shot shows a white and gray seagull perched on a gray concrete pylon in Venice, Italy. The seagull is facing left, its body angled slightly towards the camera. Its white head, neck, and chest contrast with its gray wings and back. The seagull's orange beak is slightly open. The pylon is a light gray concrete cylinder with a weathered texture. In the background, there are several buildings. To the left is a long, low, white building with a series of small, rectangular windows. In the center is a cluster of green trees. To the right is a red building with a tiled roof and several small, rectangular windows. Further right is a large, ornate, white stone building with several columns and sculptures. The sky is a pale blue, and the light is bright and clear. In the lower-left corner, there is a yellow bus with the word "Ferrovia" printed on it in black letters. There are several people walking in the background, but they are small and out of focus. |
Audio Question Answering
These models are trained to answer questions about speech in audio. Music and non-speech sounds were not part of the training data.
Inference code
messages = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"},
{"type": "text", "text": "Can you describe this audio in detail?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(
**inputs,
max_new_tokens=1000,
do_sample=False,
)
print(processor.decode(output[0], skip_special_tokens=True))
| Model | Output |
|---|---|
| E2B | This audio is a personal reflection. The speaker is talking about their final farewell address to the nation, which they delivered in Chicago. They express gratitude for the conversations they've had with the American people, noting that despite not having met them face-to-face or even greeted them, these interactions in various settings like living rooms, schools, farms, factory floors, diners, and military outposts have been what has kept them going. |
| E4B | The audio is a speech excerpt where a speaker is delivering a farewell address to the nation from Chicago. The speaker reflects on their time in office, expressing gratitude for the conversations they had with the American people across various settings like living rooms, schools, farms, factories, diners, and military outposts. The tone is reflective and appreciative, highlighting the importance of these interactions in their political journey. |
Here is an example if you want to do transcription:
Inference code
messages = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"},
{"type": "text", "text": "Transcribe the audio?"},
],
},
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
).to(model.device)
output = model.generate(
**inputs,
max_new_tokens=1000,
do_sample=False,
)
print(processor.decode(output[0], skip_special_tokens=True))
| Model | Output |
|---|---|
| E2B | This week I traveled to Chicago to deliver my final farewell address to the nation following in the tradition of presidents before me It was an opportunity to say thank you whether we've seen eye to eye or rarely agreed at all my conversations with you the American people in living rooms and schools at farms and on factory floors at diners and on distant military outposts all these conversations are what have kept me honest |
| E4B | This week I traveled to Chicago to deliver my final farewell address to the nation following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners and on distant military outposts, all these conversations are what have kept me honest. |
Multimodal Function Calling
We test the model by asking to get the weather in the place shown in the image.
Inference code
import re
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_weather",
"description": "Gets the current weather for a specific location.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "The city name"},
},
"required": ["city"],
},
},
}
tools = [WEATHER_TOOL]
messages = [
{"role": "user", "content": [
{"type": "text", "text": "What is the city in this image? Check the weather there right now."},
{"type": "image", "image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/thailand.jpg"},
]},
]
inputs = processor.apply_chat_template(
messages,
tools=[WEATHER_TOOL],
tokenize=True,
return_dict=True,
return_tensors="pt",
add_generation_prompt=True,
enable_thinking=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=1000)
input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)
print(result["content"])
| Model | Output |
|---|---|
| E2B | Analyze the Request: The user has provided an image and asked two questions:\n * "What is the city in this image?" * "Check the weather there right now."\n\n2. Analyze the Image: The image displays a large, ornate, tall structure, characteristic of temple architecture, likely in Southeast Asia (given the style). It appears to be a famous landmark. 3. Identify the Landmark/Location (Internal Knowledge/Visual Cues): The structure strongly resembles the chedis/stupas found in places like Bangkok, Thailand (e.g., Wat Arun or similar structures).\n\n4. Determine the City: Based on the visual evidence, the location is highly likely to be Bangkok, Thailand.\n\n5. Check Available Tools: The available tool is get_weather(city: STRING). 6. **Formulate the Plan:**\n * Step 1: Identify the city from the image (Bangkok).\n * Step 2: Use the get_weather tool with the identified city.\n\n7. Construct the Tool Call: get_weather(city="Bangkok")\n\n8. Generate Response: The model should first state the identified city and then call the tool to get the weather. (Self-correction: The model must call the tool in the correct format if it needs external information.)call:get_weather{city:Bangkok} |
| E4B | '1. Analyze the Request: The user has provided an image and asked two things:\n a. Identify the city in the image.\n b. Check the weather in that city right now.\n\n2. Analyze the Image: The image shows a very distinctive, ornate tower/spire structure, characteristic of Thai architecture (specifically, a chedi or prang).\n\n3. Determine City Identification (Internal Knowledge/Visual Recognition): Based on the architectural style, the structure strongly resembles prominent temples in Thailand, such as Wat Arun in Bangkok.\n\n4. Evaluate Tool Availability: I have one available tool: get_weather(city: STRING). This tool requires a city name as input.\n\n5. **Formulate a Plan:**\n a. I need the city name to use the get_weather tool.\n b. I must first identify the city from the image.\n c. Since my internal knowledge suggests Bangkok, I will proceed with that assumption, but I must acknowledge that I cannot definitively prove the location from the image alone.\n\n6. Execute Step 1 (City Identification): Assume the city is Bangkok based on the visual evidence of the architecture.\n\n7. Execute Step 2 (Tool Call Generation): Use the get_weather tool with "Bangkok" as the city argument.\n\n8. Construct the Response: Generate the appropriate tool call in JSON format.', 'answer': '<|tool_call>call:get_weather{city:<|"|>Bangkok<|"|>}<tool_call|>' |
| 31B | The user wants to know the city in the image and the current weather there. 1. Analyze the image: The image shows ornate, spire-like structures (prangs) characteristic of Thai temple architecture. Specifically, these look like the prangs at Wat Phra Kaew (Temple of the Emerald Buddha) or Wat Pho in Bangkok, Thailand. The style is very distinctive of Bangkok's royal temple complexes. 2. Identify the city: Based on the architectural style, the city is Bangkok. 3. Find the weather: I need to call the get_weather tool with city="Bangkok".call:get_weather{city:Bangkok} |
| 26/A4B | The image shows several tall, ornate towers (prangs) typical of Thai Buddhist temple architecture. Specifically, these look like the structures within the Grand Palace or Wat Phra Kaew in Bangkok, Thailand. The style is unmistakably Thai. I'll confirm this by looking at the specific towers. These are very iconic.\n2. Confirm the location: The image shows elements of the Grand Palace complex in Bangkok.\n3. Get weather information: Once the city is identified as Bangkok, I will use the get_weather tool with city='Bangkok'.call:get_weather{city:Bangkok} |
Deploy Anywhere
Gemma 4 comes with day-0 support for many open-source inference engines, and is ideal for tool calling and agents! We also release ONNX checkpoints that can run on many hardware backends, allowing use cases on edge devices or in browser!
transformers
Gemma 4 comes with first-class transformers support from the get-go 🤗. This integration allows using the model with other libraries like bitsandbytes, PEFT and TRL. Make sure to install the latest version of transformers.
pip install -U transformers
The easiest way to infer with the small Gemma 4 models is through the any-to-any pipeline. You can initialize it as follows.
from transformers import pipeline
pipe = pipeline("any-to-any", model="google/gemma-4-e2b-it")
You can then pass in images and text as follows.
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/thailand.jpg",
},
{"type": "text", "text": "Do you have travel advice going to here?"},
],
}
]
output = pipe(messages, max_new_tokens=100, return_full_text=False)
output[0]["generated_text"]
When inferring with videos, you can include the audio track using the load_audio_from_video argument.
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/rockets.mp4",
},
{"type": "text", "text": "What is happening in this video?"},
],
}
]
pipe(messages, load_audio_from_video=True)
Going a level lower, you can load Gemma 4 using the AutoModelForMultimodalLM class, especially useful for fine-tuning. The built-in chat template takes care of formatting the inputs correctly, please make sure you use it to prevent subtle mistakes when building the prompt manually.
Inference code
from transformers import AutoModelForMultimodalLM, AutoProcessor
model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-E2B-it", device_map="auto")
processor = AutoProcessor.from_pretrained("google/gemma-4-E2B-it")
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/rockets.mp4",
},
{"type": "text", "text": "What is happening in this video?"},
],
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
Llama.cpp
Gemma 4 models come with image+text support in llama.cpp from the get-go! This unlocks using Gemma 4 with all of your favorite local apps: llama-cpp server, lmstudio, Jan as well as coding agents like Pi across many backends such as Metal and CUDA.
You can install llama-cpp as follows.
curl -LsSf https://llama.app/install.sh | sh
You can then start a server compatible with the OpenAI API Replace the quantization scheme at the end of the command with the precision of your choice.
llama serve -hf ggml-org/gemma-4-E2B-it-GGUF
Check out this link for more options on combining llama.cpp with different coding agents and local apps. Find all the GGUF checkpoints in this collection.
Plug in your local agent
We worked on making sure the new models work locally with agents like openclaw, hermes, pi, and open code. All thanks to llama.cpp! Run the following to try Gemma 4 right away.
First, start your local server:
llama serve -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M
For hermes:
hermes model
For openclaw:
openclaw onboard
For pi define a ~/.pi/agent/models.json:
{
"providers": {
"llama-cpp": {
"baseUrl": "http://localhost:8080/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [
{
"id": "ggml-org-gemma-4-26b-4b-gguf"
}
]
}
}
}
For open code define a ~/.config/opencode/opencode.json:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"llama.cpp": {
"npm": "@ai-sdk/openai-compatible",
"name": "llama server (local)",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"gemma-4-26b-4b-it": {
"name": "Gemma 4 (local)",
"limit": {
"context": 128000,
"output": 8192
}
}
}
}
}
}
transformers.js
transformers.js enables running Gemma 4 right inside browser. You can check out the model card to see text-only, image & text, audio & text inference in detail here. We also shipped a demo for you to test the model here.
MLX
Full multimodal support of Gemma 4 is available using the open-source mlx-vlm library. Here's how to ask the model to describe an image:
pip install -U mlx-vlm
mlx_vlm.generate \
--model google/gemma-4-E4B-it \
--image https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg \
--prompt "Describe this image in detail"
mlx-vlm supports TurboQuant, which delivers the same accuracy as the uncompressed baseline while using ~4x less active memory and running a lot faster end-to-end. This makes long-context inference practical on Apple Silicon without sacrificing quality. Use it like this:
mlx_vlm.generate \
--model "mlx-community/gemma-4-26b-a4b-it-4bit" \
--prompt "Your prompt here" \
--kv-bits 3.5 \
--kv-quant-scheme turboquant
For audio examples and more details, please check the MLX collection.
Mistral.rs
mistral.rs is a Rust-native inference engine with day-0 Gemma 4 support across all modalities (text, image, video, audio) and builtin tool-calling and agentic functionality. Install mistral.rs:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh
irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
You can then start an OpenAI-compatible HTTP server:
mistralrs serve mistralrs-community/gemma-4-E4B-it-UQFF --from-uqff 8
Or, use interactive mode:
mistralrs run -m google/gemma-4-E4B-it --isq 8 --image image.png -i "Describe this image in detail."
mistralrs run -m google/gemma-4-E4B-it --isq 8 --audio audio.mp3 -i "Transcribe this fully."
Find all models here. Please, follow the instructions in the model cards for installation and inference guidelines.
Multi-Token Prediction Drafters
Google has released Multi-Token Prediction (MTP) drafters for the Gemma 4 family: small assistant models that accelerate inference via speculative decoding. The drafter proposes several future tokens at once, and the target model verifies them in a single forward pass. You get the same outputs as the target model, just faster — no quality loss, no changes to reasoning behaviour. Reported end-to-end speedups go up to ~3x depending on hardware, batch size, and workload.
Assistants are available for all four Gemma 4 sizes (E2B, E4B, 26B A4B, 31B). They share the KV cache with the target model to avoid recomputing context, and the smaller edge variants additionally use an embedder clustering trick to keep memory and compute low on-device.
Find the checkpoints in the Gemma 4 collection and the mlx-community collection.
DiffusionGemma: Text Generation via Diffusion
Alongside the autoregressive Gemma 4 family, Google DeepMind is releasing DiffusionGemma, a multimodal model that generates text using discrete diffusion instead of token-by-token autoregression. It's built on the same 26B A4B Mixture-of-Experts foundation (25.2B total / 3.8B active parameters, 8 active experts out of 128 plus 1 shared), takes text and image inputs, generates text, and supports up to 256K context — all under the same Apache 2.0 license.
Where a standard causal LM emits one token at a time, DiffusionGemma denoises whole blocks of tokens in parallel. The architecture is encoder-decoder: an autoregressive encoder prefills the prompt and builds the KV cache, while a decoder applies bidirectional attention over a "canvas" of 256 tokens. During multi-canvas sampling, the model iteratively denoises a full canvas with a diffusion sampler; once a canvas is finalized it's encoded and appended to the KV cache, then the next canvas begins. This block-autoregressive approach increases generation speed.
The headline benefit is throughput: parallel denoising generates roughly 15–20 tokens per forward pass, reaching per-user generation speeds exceeding 1100 tokens/second at low batch sizes (H100, FP8). Inference compute is adaptive too — simpler prompts and structured tasks like code need fewer denoising steps, so tokens-per-second scales with task complexity. It keeps the broader Gemma 4 toolkit: thinking mode, function calling, long context, native system prompts, and image understanding (OCR, document parsing, object detection, pointing) at variable aspect ratios and resolutions.
Benchmarks show an expected trade-off between speed and evaluation metrics — DiffusionGemma trails the autoregressive 26B A4B on most tasks (e.g. MMLU Pro 77.6% vs 82.6%, AIME 2026 69.1% vs 88.3%, GPQA Diamond 73.2% vs 82.3%) in exchange for its large speed advantage, while edging ahead on a few (HLE no tools 11.0% vs 8.7%).
Getting started looks just like the rest of Gemma 4, via the dedicated diffusion class:
from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor
MODEL_ID = "google/diffusiongemma-26B-A4B-it"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
MODEL_ID,
dtype="auto",
device_map="auto",
)
message = [{"role": "user", "content": "Why is the sky blue?"}]
input_ids = processor.apply_chat_template(
message,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
output = model.generate(**input_ids, max_new_tokens=512)
text = processor.decode(output[0], skip_special_tokens=False)
For best results, Google recommends the Entropy-Bounded (EB) sampler with adaptive stopping (up to 48 denoising steps, a temperature schedule decaying from 0.8 → 0.4, and an entropy bound of 0.1 for token selection). As with Gemma 4, place image content before text in your prompt, and toggle reasoning with the <|think|> control token.
Fine-tuning for all
Gemma 4 models are ideal for fine-tuning in your favorite tools and platforms and at any budget.
Fine-tuning with TRL
Gemma 4 is fully supported for fine-tuning with TRL. To celebrate, TRL has been upgraded with support for multimodal tool responses when interacting with environments, meaning models can now receive images back from tools during training, not just text.
To showcase this, we've built an example training script where Gemma 4 learns to drive in the CARLA simulator. The model sees the road through a camera, decides what to do and learns from the outcome. After training, it consistently changes lanes to avoid pedestrians. The same approach works for any task where a model needs to see and act: robotics, web browsing, or other interactive environments.
Get started:
# pip install git+https://github.com/huggingface/trl.git
python examples/grpo_carla/carla_vlm_gemma.py \
--env-urls https://sergiopaniego-carla-env.hf.space \
https://sergiopaniego-carla-env-2.hf.space \
--model google/gemma-4-E2B-it
Find the example here.
Fine-tuning with TRL on Vertex AI
Additionally, we have prepared an example on how to fine-tune Gemma 4 with TRL on Vertex AI using SFT, to showcase how to extend the function calling capabilities, whilst freezing both the vision and audio towers. The examples include how to build a custom Docker container with latest Transformers, TRL, etc. with CUDA support on Google Cloud, and how to run it via Vertex AI Serverless Training Jobs.
from google.cloud import aiplatform
aiplatform.init(
project="<PROJECT_ID>",
location="<LOCATION>",
staging_bucket="<BUCKET_URI>",
)
job = aiplatform.CustomContainerTrainingJob(
display_name="gemma-4-fine-tuning",
container_uri="<CONTAINER_URI>",
command=["python", "/gcs/gemma-4-fine-tuning/train.py"],
)
job = job.submit(
replica_count=1,
machine_type="a3-highgpu-1g",
accelerator_type="NVIDIA_H100_80GB",
accelerator_count=1,
base_output_dir="<BUCKET_URI>/output-dir",
environment_variables={
"MODEL_ID": "google/gemma-4-E2B-it",
"HF_TOKEN": <HF_TOKEN>,
},
boot_disk_size_gb=500,
)
You can find the complete example in the "Hugging Face on Google Cloud" docs at https://hf.co/docs/google-cloud/examples/vertex-ai-notebooks-fine-tune-gemma-4.
Fine-tuning with Unsloth Studio
If you want to fine tune and run a Gemma 4 model in a UI, try out Unsloth Studio. It runs locally or on Google Colab. First, install and start the app:
# install unsloth studio on MacOS, Linux, WSL
curl -fsSL https://unsloth.ai/install.sh | sh
# install unsloth studio on Windows
irm https://unsloth.ai/install.ps1 | iex
# launch unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Search for for a Gemma 4 model like google/gemma-4-E2B-it
Then select any of the Gemma 4 models from the hub.
Try Gemma 4
We have shipped demos for you to try different Gemma 4 models. We include demos based on the transformers implementation for E4B, 12B Unified, 26B/A4B MoE, and 31B dense models. There's also a WebGPU demo with transformers.js 🚀
Benchmark Results
Gemma 4 models demonstrate exceptional performance across diverse benchmarks, from reasoning and coding to vision and long-context tasks. The graph below shows model performance vs size, with Gemma 4 models forming an impressive Pareto frontier:
Source: Google (blog.google)
Here are detailed benchmark results for the instruction-tuned models:
| Benchmark | Gemma 4 31B | Gemma 4 26B A4B | Gemma 4 12B Unified | Gemma 4 E4B | Gemma 4 E2B | Gemma 3 27B (no think) |
|---|---|---|---|---|---|---|
| Reasoning & Knowledge | ||||||
| MMLU Pro | 85.2% | 82.6% | 77.2% | 69.4% | 60.0% | 67.6% |
| AIME 2026 no tools | 89.2% | 88.3% | 77.5% | 42.5% | 37.5% | 20.8% |
| GPQA Diamond | 84.3% | 82.3% | 78.8% | 58.6% | 43.4% | 42.4% |
| Tau2 (average over 3) | 76.9% | 68.2% | 69.0% | 42.2% | 24.5% | 16.2% |
| BigBench Extra Hard | 74.4% | 64.8% | 53.0% | 33.1% | 21.9% | 19.3% |
| MMMLU | 88.4% | 86.3% | 83.4% | 76.6% | 67.4% | 70.7% |
| Coding | ||||||
| LiveCodeBench v6 | 80.0% | 77.1% | 72.0% | 52.0% | 44.0% | 29.1% |
| Codeforces ELO | 2150 | 1718 | 1659 | 940 | 633 | 110 |
| HLE no tools | 19.5% | 8.7% | 5.2% | - | - | - |
| HLE with search | 26.5% | 17.2% | - | - | - | - |
| Vision | ||||||
| MMMU Pro | 76.9% | 73.8% | 69.1% | 52.6% | 44.2% | 49.7% |
| OmniDocBench 1.5 (edit distance) | 0.131 | 0.149 | 0.164 | 0.181 | 0.290 | 0.365 |
| MATH-Vision | 85.6% | 82.4% | 79.7% | 59.5% | 52.4% | 46.0% |
| MedXPertQA MM | 61.3% | 58.1% | 48.7% | 28.7% | 23.5% | - |
| Audio | ||||||
| CoVoST | - | - | 38.5* | 35.54 | 33.47 | - |
| FLEURS (lower is better) | - | - | 0.069* | 0.08 | 0.09 | - |
| Long Context | ||||||
| MRCR v2 8 needle 128k (average) | 66.4% | 44.1% | 43.4% | 25.4% | 19.1% | 13.5% |
*Excluding Chinese language.
Acknowledgements
This work wouldn't have been possible without Google's extensive contribution with the model artefact, but also the significant effort contributing the model to transformers in an effort to standardize it. The open-source ecosystem is now more complete, with a very capable, freely-licensed, open-source model. The Gemma 4 transformers integration was handled by Cyril, Raushan, Eustache, Arthur, Lysandre. We thank Joshua for the transformers.js integration and demo, Eric for mistral.rs integration, Son for Llama.cpp, Prince for MLX, Quentin, Albert and Kashif for TRL, Adarsh for SGLang transformers backend, and Toshihiro for building several demos.