继我们在发布博客中的公告之后，我们正式发布Gemma 4 12B，这是一个采用统一、无编码器架构的稠密多模态模型。
Gemma 4 12B 为本地 AI 带来了多项里程碑：
- 无编码器的多模态架构：完全绕开繁重的多阶段视觉和音频编码器，多模态数据直接输入 LLM 主干，从而降低多模态延迟。
- 我们首个支持音频输入的中等规模模型：在 Gemma 家族中，音频输入此前仅限于小型、轻量级的边缘架构（例如 E4B）。Gemma 4 12B 是首个能够原生接收音频的中等规模模型。
- 对开发者友好的尺寸：小到足以在配备 16GB VRAM 或统一内存的独立 GPU 笔记本上本地运行。为最大化本地推理速度，我们还额外发布了一个专用的多 token 预测（MTP）模型。
- 全新的 MacOS 桌面体验：我们首次发布可下载的 macOS 桌面应用程序，让开发者能够直接在消费级设备上体验完全本地的语音和视觉交互。
架构
传统多模态模型依赖冻结的、彼此独立的视觉编码器（例如 Gemma 4 在边缘尺寸上使用 150M 参数的视觉模型，在中等尺寸模型上使用 550M 参数）和音频编码器（Gemma 4 E2B 和 E4B 为 300M 参数）。在将多模态输入送入 LLM 之前，先用多个独立编码器进行处理，会导致延迟增加和内存占用碎片化。
Gemma 4 12B 通过采用单一的纯解码器 Transformer 解决了这些问题，其包含与 Gemma 4 31B Dense 模型相同的高级解码器结构。
- 视觉嵌入器（35M 参数）：取代了其他中等尺寸 Gemma 4 模型中的 27 层视觉 Transformer。原始 48x48 像素图块通过单次 matmul 投影到 LLM 隐藏维度。因子化坐标查找（X 和 Y 矩阵）将空间位置信息直接附加到输入上。
- 音频波形投影：消除了独立的音频编码器（跳过了 Gemma 4 E2B 和 E4B 中使用的 12 层 conformer）。原始 16 kHz 音频信号被切分为 40ms 帧（每帧 640 个浮点数），并线性投影到 LLM 输入空间。
- 统一微调优势：由于视觉、音频和文本输入共享完全相同的权重，你不再需要协同调优彼此独立的冻结编码器。下游适配器（例如 LoRA）或全量微调会在单次前向传播中自然地更新整个多模态 token 循环（通过 Hugging Face 或 Unsloth）。
如需更深入地了解这种无编码器架构的工作原理，请查看 Gemma 4 12B 可视化指南。
能力
Gemma 4 12B 实现了出色的性能，具备自动语音识别、智能体推理、说话人分离、视频理解、编码等多项能力。
请参阅以下示例，了解该模型智能体与多模态能力的演示：
示例 1：Gemma 4 12B 创建一个使用 Gemma 4 12B 的本地图像处理应用
得益于其智能体与多模态理解能力，Gemma 4 12B 可以轻松与 OpenCode 等现有智能体框架配合使用。在此示例中，我们使用 llama.cpp 在本地部署了它，并借助 gemma-skills 编写了一个 Gradio 应用，帮助用户处理图像。这个应用正是由构建它的同一个 Gemma 4 12B 模型驱动的！
示例 2：以 1 FPS 处理 5 分钟带音频的视频
我们使用 Gemma 4 12B 分析了 5 月 19 日 Google IO 主题演讲时刻 中的一部分，具体是 00:15:32 到 00:20:45 之间的 5 分钟。为此，我们提取了该片段的所有帧（以 1 FPS），以及视频中的提示词和音频：
提示词：
- 313 帧（1FPS，图像缩放至 视觉 token 预算 70）
"what happens when the man takes a selfie?"- 视频的音频 + 下方提示词
在这些演示视频中，当这名男子"自拍"或画面显示他把智能手机举在脸前时，这是一种巧妙的视觉化表达方式，用来直观呈现 AI 模型如何拿现有的媒体素材——比如一张个人照片或一段视频片段——对其进行"重新构想"。在这些具体的片段中，模型以一张自拍为基础，生成各种场景（比如身处空间站的人，或穿行于森林中的人）。
本质上，这名男子并不是真的在自拍；更确切地说，他是在**用表演呈现一个视觉隐喻，象征 AI 能够以某一特定输入（一张"自拍"）为基础，生成一整个全新的内容世界。** 这是 Gemini Omni 模型"Swap"和"Build worlds"演示的一部分，展示了它执行复杂多模态推理与创意生成的能力。
端侧与桌面端服务：由 LiteRT-LM 驱动
配合 Gemma 4 12B 的发布，我们正式推出由 LiteRT-LM 驱动的强大端侧开发者集成，将零延迟的本地 AI 执行能力原生带入标准桌面环境：
1. 原生 MacOS 应用：移动端 Google AI Edge Gallery 正式扩展到桌面平台，在 Apple Silicon GPU 上原生离线运行 Gemma 4 12B。它配备了一个安全的沙箱化 Python 执行循环，可在聊天气泡内编写、执行并绘制科学图表。与此同时，Mac 上的 Google AI Edge Eloquent 应用开始支持 Gemma 12B，为 Voice Edit 对话式输入提供支持。
2. 即插即用的本地 API 服务器（litert-lm serve）：使用新的 litert-lm serve CLI 命令。将 Gemma 4 12B 作为本地、兼容 OpenAI 的 API 服务器运行。无缝接入标准集成（例如 Continue、Aider、OpenClaw、Hermes 或 OpenCode），利用内存中的无状态前缀缓存来匹配上下文历史，并即时绕过预填充延迟。
litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm gemma-4-12B-it.litertlm gemma4-12b
# Start the OpenAI-compatible server
litert-lm serve在 Google AI Edge Gallery 博客上可找到关于它的深度解析。
即刻开始
准备好用 Gemma 家族首个无编码器架构构建本地多模态智能体了吗？以下是你可以立即上手的方式
- 亲自试一试：在 LM Studio、Ollama、Google AI Edge Gallery App、Google AI Edge Eloquent 应用以及 LiteRT-LM CLI 中，只需点击几下即可上手体验。
- 下载权重：直接从 Hugging Face 和 Kaggle 下载预训练和指令微调的 checkpoint。
- 集成与学习：查阅开发者文档和快速入门 notebook。
- 使用你喜爱的开发工具：用 Hugging Face Transformers、llama.cpp、MLX、SGLang 和 vLLM 实现本地推理流水线，或使用 Unsloth 高效地进行微调。
- 用 Gemma Skills 解锁智能体开发：为支持智能体基于最新的 Gemma 进展进行构建，我们发布了官方技能仓库。这是一个专为让智能体能够使用 Gemma 模型进行构建而设计的技能库。
- 按你的方式部署： 使用 Google Cloud 在生产环境中启动端点。通过 Gemini Enterprise Agent Platform Model Garden、Cloud Run 和 GKE，按你的方式部署。
Following the announcement in our launch blog, we are releasing Gemma 4 12B, a dense multimodal model with a unified, encoder-free architecture.
Gemma 4 12B introduces several milestones for local AI:
- A multimodal encoder-free architecture: Bypassing heavy multi-stage vision and audio encoders entirely, multimodal data is fed straight into the LLM backbone, reducing multimodal latency.
- Our first medium-sized model with audio input: In the Gemma family, audio inputs were restricted to small, lightweight edge architectures (e.g. E4B). Gemma 4 12B is the first medium-sized model capable of natively ingesting audio.
- Developer-friendly size: Small enough to run locally on dedicated GPU laptops with 16GB VRAM or unified memory. To maximize local inference speeds, we are additionally releasing a dedicated multi-token prediction (MTP) model.
- New MacOS desktop experience: For the first time, we are releasing downloadable macOS desktop applications, letting developers experience fully local spoken and visual interaction directly on consumer-grade devices.
The Architecture
Traditional multimodal models rely on frozen, separate vision encoders (e.g., Gemma 4 uses a 150M parameter vision model for edge sizes and 550M for medium-sized models) and audio encoders (300M parameters for Gemma 4 E2B and E4B). Processing multimodal inputs with multiple separate encoders before feeding them to the LLM leads to increased latency and fragmented memory footprints.
Gemma 4 12B solves these issues by utilizing a single decoder-only transformer containing the same advanced decoder structure as the Gemma 4 31B Dense model.
- Vision embedder (35M parameters): Replaces the 27 vision transformer layers of the other medium-sized Gemma 4 models. Raw 48x48 pixel patches are projected to the LLM hidden dimension with a single matmul. A factorized coordinate lookup (X and Y matrices) attaches spatial location information directly to the input.
- Audio wave projection: Eliminates the separate audio encoder (skipping the 12 conformer layers used in Gemma 4 E2B and E4B). Raw 16 kHz audio signals are sliced into 40ms frames (640 floats each) and projected linearly to the LLM input space.
- Unified fine-tuning advantage: Because vision, audio, and text inputs share the exact same weights, you no longer have to co-tune separate frozen encoders. Downstream adapter (e.g. LoRA) or full tuning naturally update the entire multimodal token loop in a single pass (via Hugging Face or Unsloth).
For a more in-depth overview of how this encoder-free architecture works, check out A Visual Guide to Gemma 4 12B.
Capabilities
Gemma 4 12B achieves outstanding performance, with capabilities such as automatic speech recognition, agentic reasoning, diarization, video understanding, coding, and more.
See below examples for a demonstration of the model's agentic and multimodal capabilities:
Example 1: Gemma 4 12B creates a local image processing app that uses Gemma 4 12B
Thanks to its agentic and multimodal understanding capabilities, Gemma 4 12B can be easily used with existing agent harnesses like OpenCode. In this example, we served it locally using llama.cpp using the gemma-skills to code a Gradio app that helped the user process images. This app was powered by the same Gemma 4 12B model that built it!
Example 2: Processing 5 minutes of Video at 1 FPS with audio
We used Gemma 4 12B to analyse a part from the Google IO Keynote moment from May 19, specifically the 5 minutes between 00:15:32 and 00:20:45. To do that, we extracted all the frames of that segment (at 1 FPS), plus the prompt and the audio from the video:
Prompt:
- 313 frames (at 1FPS, images resized to visual token budget 70)
"what happens when the man takes a selfie?"- the Video's audio + the prompt below
When the man takes a "selfie" or is shown holding a smartphone in front of his face in these demonstration videos, it is a clever way to visually represent how the AI models can take existing media—like a personal photo or a video clip—and "reimagine" it. In these specific clips, the model is taking a selfie and using it as a base to generate various scenarios (like a person in a space station, or walking through a forest).
Essentially, the man is not actually taking a selfie; rather, he is **acting out a visual metaphor for the AI's capability to take one specific input (a "selfie") and generate a whole world of new content based on it.** This is part of the "Swap" and "Build worlds" demonstrations of the Gemini Omni model, showing its ability to perform complex, multi-modal reasoning and creative generation.
On-Device & Desktop Serving: Powered by LiteRT-LM
In tandem with the Gemma 4 12B launch, we are officially introducing powerful on-device developer integrations powered by LiteRT-LM, bringing zero-latency local AI execution natively to standard desktop environments:
1.Native MacOS Apps: The mobile Google AI Edge Gallery is officially expanding to desktop platforms, running Gemma 4 12B offline, natively on Apple Silicon GPUs. It comes with a secure sandboxed Python execution loop to write, execute, and plot scientific charts inside the chat bubble. In parallel, the Google AI Edge Eloquent app on Mac launches support for Gemma 12B to power Voice Edit conversational inputs.
2. Drop-in Local API Servers (litert-lm serve): Run Gemma 4 12B as a local, OpenAI-compatible API server using the new litert-lm serve CLI command. Seamlessly connect standard integrations (e.g., Continue, Aider, OpenClaw, Hermes or OpenCode), leveraging stateless prefix caching in memory to match context history and instantly bypass prefill latency.
litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm gemma-4-12B-it.litertlm gemma4-12b
# Start the OpenAI-compatible server
litert-lm serveFind a deep dive about it on the Google AI Edge Gallery blog.
Getting Started Today
Ready to build local multimodal agents with the first encoder-free architecture of the Gemma family? Here is how you can jump in today
- Try it yourself: Experiment with a couple of clicks in LM Studio, Ollama, Google AI Edge Gallery App, the Google AI Edge Eloquent app and the LiteRT-LM CLI
- Download the weights: Download the pre-trained and instruction-tuned checkpoints directly from Hugging Face and Kaggle.
- Integrate & learn: Review the developer documentation and the quick start notebook.
- Use your favorite development tools: Implement local inference pipelines with Hugging Face Transformers, llama.cpp, MLX, SGLang, and vLLM, or fine-tune with efficiency using Unsloth.
- Unlock Agentic Development with Gemma Skills: To support agents to build with the latest Gemma advancements, we are releasing our official Skills Repository. This is a library of skills designed specifically to enable agents to build with Gemma models.
- Deploy your way: Spin up endpoints in production using Google Cloud. Deploy your way through Gemini Enterprise Agent Platform Model Garden, Cloud Run and GKE.