# Gemini Embedding 2：来自Gemini的原生多模态嵌入模型 - 来源：HuggingFace Daily Papers（社区热门论文） - 发布时间：2026-05-26 08:00 - AIHOT 分数：72 - AIHOT 标记：精选 - AIHOT 链接：https://aihot.news/items/cmpnwyk2c0104slv4ynvin2w3 - 原文链接：https://arxiv.org/abs/2605.27295 ## 精选理由 Google 把多模态嵌入统一到一个模型里了，文本、代码、跨模态检索全面刷榜，做 RAG 和搜索的该认真看看了。 ## AI 摘要 Google DeepMind推出Gemini Embedding 2，这是一款原生多模态嵌入模型，支持在统一表示空间中嵌入视频、音频、图像和文本。该模型利用Gemini的多模态能力，通过大规模对比学习实现SOTA性能。在关键基准上表现优异：MSCOCO取得62.9 R@1，Vatex取得68.8 NDCG@10，MTEB multilingual达到69.9，MTEB Code达到84.0，超越了专用模型。其统一能力使其适用于RAG、推荐与搜索等下游任务，并在天文学、生物科学、艺术和烹饪等专业领域展现出强大的零样本性能。 ## 正文 Madhuri Shanbhogue Zhe Li Shanfeng Zhang Gustavo Hernández Ábrego Shih-Cheng Huang Aashi Jain Daniel Salz Sonam Goenka Chaitra Hegde Ji Ma Feiyang Chen Jiaxing Wu Tanmaya Dabral Babak Samari Kevin Poulet Daniel Cer Kaifeng Chen Paul Suganathan Hui Hui Jovan Andonov Philippe Schlattner Jay Han Iftekhar Naim Wing Lowe Vladimir Pchelin Albert Yang Yi-Ting Chen Zhongli Ding Grace Zhang Georg Heigold Yichang Chen Antoine Reveillon Brendan Mccloskey Wenlei Zhou Dahun Kim Rui Meng Emma Wang Jack Zheng Halley Fede Zhen Yang Keegan Mosley Brian Potetz Sahil Dua Henrique Schechter Vera Shen Gao Hesen Zhang Andreas Hess Hengxuan Ying Alberto Montes Karan Gill Min Choi Sebastian Russo Anja Hauth Jinhyuk Lee Michael Boratko Megan Barnes Vikram Rao Claudiu Musat Cyril Allauzen Ehsan Variani Shankar Kumar Tom Bagby Junyi Jiao Yang Gu Tengxin Li Ayush Agrawal Roberto Santana Dev Nath Stephen Karukas Shuoxuan Han Lucia Loher Alice Twu Nidhi Vyas Siddharth Bhai Frank Palma Gomez Wangyuan Zhang Chaoren Liu Jizheng Yang Steve Qiu Shijie Zhang Sujay Kulkarni Sascha Rothe Sean Nakamoto Raphael Hoffmann Zach Gleicher Yunhsuan Sung Qin Yin Tom Duerig Mojtaba Seyedhosseini Abstract We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields – from astronomy and bioscience to fine arts and the culinary arts – establishes it as a highly reliable, out-of-the-box representation even for specialized domains. \undefine@key newfloatplacement\undefine@keynewfloatname\undefine@keynewfloatfileext\undefine@keynewfloatwithin 1 Introduction Embedding models provide dense vector representations capturing semantic information that is crucial for adaptation in a wide range of downstream tasks. With foundational models being natively multimodal and powered with exceptionally growing capabilities, it is important to ensure embedding models capture semantic information within and across all modalities in a coherent manner. Such general-purpose embedding models will also enhance the performance across a broad spectrum of applications like video recommendations and document search which are rich in information across different modalities but since the contained modalities are not inherently homogenous, they can benefit from having rich semantic information from across all modalities. Existing multimodal embedding models like CLIP [radford2021learning], ALIGN [jia2021scaling], SigLIP 2 [tschannen2025siglip], CoCa [yu2022cocacontrastivecaptionersimagetext] embed heterogenous modalities by using paired cross-modal data and training modality-specific encoders to encode them into a unified vector space. This late-fusion approach results in good unimodal and cross-modal capabilities but has a key limitation in handling mixed-modality inputs and lacks richness since it does not utilize interactions between modalities. With advances in Multimodal Large Language Models (MLLMs), it is now possible to achieve semantically richer embeddings enabled by the deep fusion of cross-modal interactions. Figure 1: Conceptual overview of the Gemini Embedding 2 workflow. The model natively processes heterogeneous inputs—text, images, video, audio, documents, and their combinations—mapping them into a single, unified high-dimensional vector space where cross-modal semantic relationships are preserved. In this work, we introduce a generalizable multimodal embedding model that embeds video, audio, image, text modalities, and any arbitrary combination thereof into a single representation space. The multimodal Gemini Embedding 2 is trained by leveraging Gemini’s [comanici2025gemini] capabilities and utilizing multi-task training with a diverse set of tasks resulting in a model that captures various interactions between modalities. Figure˜1 shows a high-level representation of how multimodal Gemini Embedding 2 maps the heterogenous sources into a unified vector space. The curated set of tasks help the model generalize across a wide variety of enterprise use cases like document retrieval, video recommendation, audio-based search, and RAG applications [lewis2020retrieval]. Crucially, enabling the model to handle interleaved sequences of images, text, and video facilitates complex, novel retrieval paradigms—such as zeroing in on specific temporal events in a video using combined visual and textual prompts. Using Gemini’s capabilities we also show that native audio understanding and native multimodal understanding outperforms text-based alternatives like ASR or captioning. Figure 2: Gemini Embedding 2 shows strong performance across multimodal retrieval tasks spanning image, text, video, and document modalities. ∗MTEB number is reported for Voyage-3.5 since Voyage-3.5-multimodal does not report MTEB. We evaluate comprehensively on a wide variety of benchmarks, both academic-focused and enterprise-focused. As shown in Figure˜2, our model achieves state-of-the-art performance compared to other models. For evaluating the text embedding capabilities, we rely on the Massive Multilingual Text Embedding Benchmark (MMTEB) [enevoldsen2025mmteb] which consists of multi-lingual tasks spanning key downstream embedding use cases like retrieval, clustering, classification, etc. Gemini Embedding 2 achieves state-of-the-art performance on multilingual and code surpassing existing models on the leaderboard. We demonstrate strong numbers on a broad range of cross-modal retrieval benchmarks like MSCOCO [chen2015microsoftcococaptionsdata], Flickr30k [plummer2016flickr30kentitiescollectingregiontophrase], and MSR-VTT [Xu2016MSRVTTAL]. We also demonstrate the model’s ability to generalize to most multimodal retrieval tasks in general as well as specialized domains. 2 Related Work Large Language Models as Text Embedders The paradigm of text embedding models has matured from relying on purely encoder-only architectures (e.g., BERT [devlin2019bert], RoBERTa [liu2019robertarobustlyoptimizedbert]) to utilizing decoder-only or massive LLM backbones. Models such as the BGE [chen2025m3embeddingmultilingualitymultifunctionalitymultigranularity] series and E5 [wang2024textembeddingsweaklysupervisedcontrastive] established instruction-tuned representations, effectively unifying downstream tasks—like semantic search, clustering, and classification—into a single model via task-specific prefixes. Recognizing the rich semantic understanding capabilities of LLMs, recent research has focused heavily on LLM-augmented training and distillation. The Gecko model [lee2024geckoversatiletextembeddings] demonstrated that lightweight, highly-efficient retrievers can be trained through a two-step distillation pipeline that leverages the vast knowledge of massive LLM teachers. Concurrently, NV-Embed [lee2025nvembedimprovedtechniquestraining] achieved strong performance on the MMTEB leaderboard [muennighoff2023mteb] by transforming decoder-only LLMs into generalist embedders using instruction-tuned contrastive learning and the aggressive integration of synthetic, non-retrieval data. Gemini Embedding [lee2025geminiembeddinggeneralizableembeddings] demonstrated state-of-the-art performance on the MMTEB leaderboard due to utilizing synthetic data and excellent generalization to multilingual tasks through the powerful pre-training of Gemini. Evolution of Multimodal Embedders Early multimodal embedding paradigms, exemplified by dual-tower models like CLIP [radford2021learning] and ALIGN [jia2021scaling], were limited by their reliance on narrow contrastive learning objectives over simple image–text pairs. Today, the field is gravitating towards multimodal architectures capable of mapping text, code, images, structured documents, audio, and video into a single, unified, continuous semantic space. Embedding models are trained by extending existing MLLMs for retrieval via multi-stage contrastive training thereby enabling excellent cross-modal retrieval capabilities. SAIL-Embedding [lin2025sailembeddingtechnicalreportomnimodal] further illustrates this shift by employing a content-aware progressive training methodology mapping multimodal representations seamlessly into industrial recommendation environments (e.g., sequence-to-item prediction). Similarly, Amazon Nova MME [AWS2025novaembeddings] and SigLIP 2 [tschannen2025siglip] have demonstrated strong performance in unifying disparate modalities for cross-modal retrieval workflows. Architectural Adaptations for Bidirectional Attention While causal (autoregressive) LLMs excel in generative tasks, their inherently unidirectional attention mechanism imposes unnecessary limits when generating dense, context-aware embeddings. Several innovative frameworks have emerged to circumvent this limitation. MoCa [chen2025mocamodalityawarecontinualpretraining] directly addresses this by introducing modality-aware continual pre-training, utilizing a joint reconstruction objective that denoises interleaved text and image inputs to force bidirectional context-aware reasoning on top of a causal backbone. Similarly, MM-Embed [lin2025mmembeduniversalmultimodalretrieval] tackles the problem of modality bias through modality-aware hard negative mining, ensuring that embedding models do not disproportionally favor text-to-text resonance at the expense of cross-modal relevance. Adaptation to Enterprise Use Cases With enterprise and agentic needs scaling to massive contexts and increasingly focused on documents, modern embedders are required to ingest vast informational payloads efficiently. Models utilize specialized visual-document processing (such as tiled mixtures of vision encoders) to embed complex PDFs, charts, and tables which causes the RAG system’s quality to be dependent on various parts of the processing pipeline like chunking strategies etc. While these preceding architectures have successfully pushed the boundaries of multi-stage distillation, LLM backbone adaptation, and applications to enterprise use cases, they predominantly address these axes in isolation. Gemini Embedding 2 unifies these capabilities into a single model that spans a breadth of use cases across which the model can be used out-of-the-box. 3 Multimodal Gemini Embedding In this section we provide technical details of the Multimodal Gemini Embedding 2 in terms of the model architecture, the objective function, and the training recipe. 3.1 Model Architecture The Gemini Embedding 2 model is built to create holistic representations of inputs of different modalities and of inputs that combine such modalities. These representations can be used in diverse downstream tasks including retrieval, clustering, classification, and ranking. Gemini Embedding 2 leverages the multimodal and cross-modal power of Gemini to build such representations. The embedding model is initialized from Gemini and further fine-tuned with task-specific, modality-specific, and cross-modality training. This allows Gemini Embedding 2 to build representations on top of the vast knowledge already present in the Gemini parameters. In this sense, initializing Gemini Embedding 2 from Gemini can be understood as the “pre-training" stage of the embedding model. Gemini Embedding 2 constructs representations in a manner similar to our previous Gemini Embedding model [lee2025geminiembeddinggeneralizableembeddings], but with the important difference that different modalities require different steps to convert the raw format into a sequence of tokens. In Gemini Embedding 2 we leverage Gemini to do these types of data and format conversions. In this way, the model can take as input raw images, video or audio in the formats natively supported by Gemini. After tokenization, an input sequence of tokens is processed by , a transformer with bidirectional attention initialized from Gemini, producing a sequence of token embeddings , where is the transformer model dimension. To generate a single embedding representing all the information in the input, a pooler is applied, . Prior research [suganthan2025adaptingdecoder] demonstrated that simple pooling strategies can be effective in model adaptation. Therefore we choose mean pooling, and simply average the token embeddings along the sequence axis. Finally, a randomly initialized linear projection is applied to scale the embedding to the target dimension, , where is the output embedding dimension. 3.2 Training Objective The multimodal nature of Gemini Embedding 2 requires a multi-task and multi-stage type of training. This way different modalities can be trained in separate tasks. We used a multitude of single-modality tasks, multimodal tasks, as well as cross-modal tasks. Similar to our previous version [lee2025geminiembeddinggeneralizableembeddings], the multimodal Gemini Embedding 2 model was trained with a noise-contrastive estimation (NCE) loss with in-batch negatives [oord2018representation]. The exact loss differs slightly depending on the task being trained. In general, a training example includes a query , a positive target and (optionally) a hard negative target . In text-only training tasks, each example also has a prescribed task string , for example "question answering" or "fact checking", describing the nature of the task. During training, we randomly drop off the task string to augment the robustness of the model to different modality inputs where the task strings are not used. The query and passages are embedded as vectors in : (1) Given a batch of size the loss applied to these embeddings is as follows: (2) where is cosine similarity, and (3) This masking term is particularly relevant for classification tasks, where the number of targets (labels) is small. It should be noted that the second term in the denominator is omitted if no hard negatives are provided. In order to support different dimensions of embeddings with a single model, we adapt the above loss using MRL [kusupati2022matryoshka] into separate losses across overlapping sub-dimensions of the embedding dimensions (e.g. multi-loss training with one loss for the first 768 embedding dimensions, another for the first 1,536 dimensions, and so on). Gemini Embedding 2 provides dimensional embeddings, with the MRL support optimized for 768 and 1,536 dimensions. 3.3 Recipe We heavily lean on the multi-task nature of our training setup to let the model learn from each of the different tasks that, as mentioned in section Section˜3.2, contribute in different ways to build the unified embedding space across the different modalities. We adopt the multi-stage training from previous models like Gecko [lee2024geckoversatiletextembeddings] and Gemini Embedding [lee2025geminiembeddinggeneralizableembeddings] as described below. Pre-Fine-Tuning (PFT) To adapt the parameters in the model from auto-regressive generation to encoding, this stage uses as training a large number of potentially noisy query–target pairs in a multi-task setup. Further, in this stage we find it beneficial to use large batch sizes which provide more stable gradients, mitigating the impact of the noisy inputs. During this stage, only image, text and code tasks are used in our multi-task setup. The examples from each different task are sampled at pre-specified sampling rates to build training batches of a single task. Fine-Tuning (FT) The fine-tuning stage for this model is based on training with a large number of text, code, document, image, audio, and video tasks. Many, but not all, of the tasks in this fine-tuning include examples that contain query, target, and hard negative target triplets. For this training stage we found it beneficial to tune batch sizes for each task to improve quality on corresponding evaluations. In this stage we also sample examples from one single task to build the training batches. The alignment between modalities is based on training multiple single-modality batches as well as cross-modality ones. As in the previous stage, training with all the different tasks and modalities require a multi-task training setup and the sampling rates of each of the different tasks are defined empirically. Empirically, we found that balancing overall performance across all modalities was sensitive to hyper-parameters like sampling rates and batch sizes in the multi-task setup. Model Soup To systematize the combination of different checkpoints and obtain additional generalization performance across the different modalities, we average the parameters obtained from individual fine-tuning runs. We experimented with different combinations of parameters, including averaging checkpoints from the same training run [izmailov2018averaging], from different training runs [wortsman2022model], as well as various weighted averages. 4 Evaluation We rigorously evaluate Gemini Embedding 2 across a comprehensive suite of multimodal and unimodal benchmarks, demonstrating its state-of-the-art capabilities in text, image, video, and audio understanding. Unlike competing models that often rely on brittle, task-specific instructions, Gemini Embedding 2 provides a robust, unified latent space that delivers high performance in zero-shot settings without the need for manual prompt engineering. 4.1 Multimodal Retrieval 媒体内容 · 前往原文查看 Table 1: Comparison of embedding models on retrieval benchmarks. Our model shows strong performance across a variety of unimodal, cross-modal, and multimodal retrieval tasks. †: Average over intersection of tasks where the metrics are available for all models. Modality abbreviations: V=Video, A=Audio, I=Image, T=Text. ‡: Reported by accessing available APIs unless self-reported. Gemini Amazon Nova‡ Voyage-3.5- ‡ multimodal‡ Embedding 2 MME multimodal embedding@001 Legacy Google model Image Image (Recall@1) GUIEC [guiec] 79.4 68.6 69.4 69.5 ImageNet [5206848] 83.6 - - 71.8 Text Image (Recall@1) Mean† 80.5 71.6 75.8 69.5 MSCOCO [chen2015microsoftcococaptionsdata] 62.9 57.2 58.1 53.1 Flickr30k [plummer2016flickr30kentitiescollectingregiontophrase] 89.1 81.6 89.9 81.4 DOCCI [DOCCI] 93.4 84.0 83.8 - TextCaps [TextCaps] 89.6 76.0 79.4 74.0 Image Text (Recall@1) Mean† 91.2 81.6 85.9 83.4 MSCOCO [chen2015microsoftcococaptionsdata] 78.8 68.3 74.5 68.2 Flickr30k [plummer2016flickr30kentitiescollectingregiontophrase] 97.4 87.5 94.5 94.0 DOCCI [DOCCI] 91.3 76.5 77.4 - TextCaps [TextCaps] 97.4 88.9 88.6 88.1 Text Video (NDCG@10) Mean 63.1 54.0 49.9 49.2 Vatex [wang2020vatexlargescalehighqualitymultilingual] 68.8 60.3 55.2 54.9 MSR-VTT [Xu2016MSRVTTAL] 68.0 67.0 63.0 57.9 YouCook2 [zhou2017automaticlearningproceduresweb] 52.5 34.7 31.4 34.9 Image+Text Text (Recall@20) EncyclopedicVQA [Mensink_2023_ICCV] 71.5 - 58.6 - Document Retrieval (NDCG@10) ViDoRe V2 [mace2025vidorebenchmarkv2raising] 64.9 60.6 65.5 28.9 Overall Performance† 77.2 68.2 70.0 64.1 Modality V/A/I/T V/A/I/T V/I/T I/T We evaluate Gemini Embedding 2 against other multimodal embedding models — Voyage-3.5-multimodal [VoyageAI2026multimodal35], Amazon Nova MME [AWS2025novaembeddings], and Google’s legacy model multimodalembedding@001 [google_cloud_multimodal_embeddings] — across a diverse suite of unimodal, cross-modal and multimodal retrieval benchmarks spanning image, text, and video modalities (see Table˜1). For unimodal image evaluation, we utilize the Google Universal Embedding Challenge (GUIEC) [araujo2022google] which requires instance-level retrieval over a large-sized index consisting of 200,000 images. We also evaluate cross-modal retrieval quality on image-to-text and text-to-image benchmarks including MSCOCO [chen2015microsoftcococaptionsdata], Flickr30K [plummer2016flickr30kentitiescollectingregiontophrase], DOCCI [DOCCI] and TextCaps [TextCaps]. These tasks range from challenging the models on basic image captioning to long captions including spatial reasoning and scene text understanding. We embed the images and texts separately using Gemini Embedding 2 and then retrieve using cosine similarity between queries and documents over the whole test set. We also evaluate on multimodal embedding capabilities by embedding images and texts together. We do visual question answering as a retrieval evaluation using EncyclopedicVQA [Mensink_2023_ICCV] where we embed the image along with the question to retrieve the correct answer. For text-to-video retrieval, we evaluate on Vatex [wang2020vatexlargescalehighqualitymultilingual], MSR-VTT [Xu_2016_CVPR], and YouCook2 [zhou2017automaticlearningproceduresweb] where the video is embedded at 1 FPS up to 32 frames. Gemini Embedding 2 achieves the highest global mean score and leads decisively on unimodal image retrieval, text-to-image, image-to-text, and text-to-video tasks, with particularly strong results on long-caption benchmarks such as DOCCI and TextCaps. The training mixture shows very good capabilities to generalize to third-party evaluation tasks like Vatex, MSR-VTT, and YouCook2 despite not including any specific in-domain training splits of those datasets. On the ViDoRe Benchmark V2 [mace2025vidorebenchmarkv2raising] document retrieval benchmark, as presented in Table˜1 Gemini Embedding 2 achieves a score of 64.9, delivering competitive performance in a task that demands understanding of page-level visual structure, layout, and embedded text. This places Gemini Embedding 2 ahead of Amazon Nova MME (60.6) and within close range of Voyage-3.5-multimodal (65.5). Gemini Embedding 2 also stands out as one of only two models in this comparison to support the full Video/Audio/Image/Text modality set (alongside Amazon Nova MME), making its document retrieval performance particularly noteworthy given the breadth of tasks it is simultaneously optimized for. 4.2 MMTEB 媒体内容 · 前往原文查看 Table 2: Comparison of multimodal and text-only embedding models on the Massive Text Embedding Benchmark, MTEB(Multilingual), MTEB Code v1, and CoIR benchmarks. Modality abbreviations: V=Video, A=Audio, I=Image, T=Text. ∗: only self-reported the aggregated MTEB(Multilingual) mean score. †: voyage-3.5 for MTEB(Multilingual) and voyage-code-3 in CoIR. ‡: Results were not reported. Gemini Amazon Nova∗ Gemini voyage-3.5/† Embedding 2 MME Embedding voyage-code-3 MTEB(Multilingual) [enevoldsen2025mmteb] Mean (Task) 69.9 63.8 68.4 58.5 Mean (Type) 61.2 59.6 51.9 - Bitext Mining 85.4 79.3 60.5 - Classification 73.1 71.8 58.5 - Clustering 55.3 54.6 45.9 - Inst. Retrieval 2.9 5.2 6.5 - Multilabel Class. 32.2 29.2 21.7 - Pair Class. 83.2 83.6 76.0 - Reranking 69.0 65.7 64.2 - Retrieval 70.0 67.7 64.0 - STS 79.4 79.4 70.0 MTEB(Code) [enevoldsen2025mmteb] Mean 84.0 –‡ 76.0 –‡ CoIR [li2024coircomprehensivebenchmarkcode] Mean 82.3 –‡ 73.9 78.5 Modality V/A/I/T V/A/I/T T T The multilingual benchmark MMTEB [enevoldsen2025mmteb] consists of a large collection of individual evaluation tasks covering 250+ languages and 10 task types: Bitext Mining, Classification, Clustering, Instruction Retrieval, Multilabel Classification, Pair Classification, Reranking, Retrieval, STS, and Summarization. Gemini Embedding 2 overall performance, along with the performance of other multimodal models, is presented in Table˜2 where we also include the modalities supported by each model. The MMTEB results demonstrate that Gemini Embedding 2 outperforms other multimodal models on this text-only benchmark, indicating that its expanded multimodal capabilities do not compromise its performance on purely textual tasks. Relative to our previous text-only Gemini Embedding model, the new multimodal Gemini Embedding 2 shows stronger performance surpassing the Mean (by task) of 68.32 of our previous model with an equivalent of 69.9. Moreover, our multimodal Gemini Embedding 2 sets a new state-of-the-art performance level in task-specific evaluations such as MTEB Code v1 [enevoldsen2025mmteb], which consists of 12 code retrieval tasks in 15 coding languages, and the Code Information Retrieval benchmark, CoIR [li2024coircomprehensivebenchmarkcode], which includes 10 of coding retrieval tasks in 9 coding languages. Table˜2 also shows that our new Gemini Embedding 2 model achieves performance that is considerably better in these benchmarks than our previous Gemini Embedding text-only model. Notably, Gemini Embedding 2 is also considerably better relative to other text-only models and also better than domain-specific models such as voyage-code-3. 4.3 MSEB 媒体内容 · 前往原文查看 Table 3: Results on the passage retrieval split of the MSEB benchmark. Utilizing native audio input consistently enhances retrieval performance over the ASR baseline, yielding robust gains in both intra-lingual and cross-lingual generalization setups. Model Setup Average Retrieval Split (mrr@10) Passage In-Lang Passage Cross-Lang Gemini Embedding 2 w/ ASR 70.40 73.58 67.55 Gemini Embedding 2 w/ Native Audio 73.99 75.58 72.56 To rigorously evaluate the auditory capabilities of Gemini Embedding 2, we benchmark the model on the Massive Sound Embedding Benchmark (MSEB) [heigold2026massivesoundembeddingbenchmark]. We focus our evaluation on the retrieval split of MSEB. The model is given a spoken query and the task is to find the most relevant information for the query in a large corpus of text documents. 4.3.1 Experimental Setup A persistent challenge in multimodal retrieval is the bottleneck introduced by standard pipelined approaches, where audio is typically transcribed to text before producing the embeddings. To isolate the impact of our unified multimodal architecture, we juxtapose two distinct input modalities: Gemini Embedding 2 with ASR: A cascaded baseline where the raw audio signal is first transcribed into text via an Automatic Speech Recognition (ASR) system, and the resulting text is subsequently encoded. Gemini Embedding 2 with audio: Our proposed approach, which directly processes raw audio inputs without intermediate textual transcription. We utilize Mean Reciprocal Rank at 10 (mrr@10) as our principal evaluation metric. The retrieval setup is further stratified into two key partitions to assess generalization: PassageInLang (intra-lingual retrieval within the same language) and PassageCrossLang (cross-lingual retrieval). 4.3.2 Results As shown in Table˜3, the results demonstrate that utilizing native audio processing significantly enhances retrieval performance over the ASR baseline. As shown, Gemini Embedding 2 with native audio achieves an average retrieval mrr@10 of 73.99, yielding a substantial improvement over the ASR-based approach (70.40). Breaking down the task partitions, we observe consistent gains across varying degrees of linguistic complexity: PassageInLang: Direct audio modeling improves same-language retrieval by +2.0 points (75.58 vs. 73.58). The performance gap between the cascade baseline and Gemini Embedding 2 highlights a structural flaw in pipeline architectures. The cascade system (ASR → Retrieval) in this experiment—suffers heavily from error propagation. If the ASR system misinterprets an ambiguous audio snippet and commits to an incorrect text output, the downstream retrieval system faces a fundamentally altered query, leading to poor search results. Gemini Embedding 2 overcomes this bottleneck by natively encoding the raw audio directly. Instead of forcing a "hard" textual decision (e.g., "recognize speech" vs. "wreck a nice beach"), the resulting embedding preserves the inherent ambiguity of the original acoustic signal. This robust, continuous representation gives the system a significantly better chance of surfacing the correct retrieval results by preserving rich acoustic cues (e.g., prosody, intonation, and emphasis). PassageCrossLang: Notably, the performance delta widens in cross-lingual setups. Native audio embeddings yield a striking +5.01 point enhancement (72.56 vs. 67.55). The dramatic jump in PassageCrossLang validates that the modality-agnostic latent space of Gemini Embedding 2 deeply aligns semantic features regardless of the source audio’s spoken language, generalizing robustly beyond the strict phonetic bounds parameterized by an intermediate ASR transcriber. In aggregate, the MSEB benchmark corroborates that Gemini Embedding 2 successfully models contiguous raw audio, effectively consolidating a holistic representation that significantly outperforms transcription-reliant bottlenecks. 5 Ablation Study To better understand how Gemini Embedding 2 achieves great performance across many different tasks and languages, we provide a systematic analysis of our training recipe. 媒体内容 · 前往原文查看 Table 4: Image-to-Text Retrieval (R@5) performance across various Specialized Domains. Model Variant MicroVQA [burgess2025microvqa] ArtCap [lu2022artcap] AstroLLaVA [zaman2025astrollava] Recipe1M [marin2021recipe1m+] Ingredients Instructions CLIP [radford2021learning] Base Patch32 34.1 34.1 21.2 64.6 61.1 Large Patch14 44.4 49.4 28.8 76.5 74.6 Large Patch14-336 46.7 52.2 31.6 76.0 75.6 ALIGN [jia2021scaling] Base 48.1 49.2 18.4 70.3 70.8 SigLIP 2 [tschannen2025siglip] Base Patch16-256 23.0 16.3 6.3 69.8 70.7 Large Patch16-384 27.4 7.3 11.0 78.7 78.3 Giant Patch16-384 33.3 8.4 13.2 81.2 80.4 TIPS [maninis2025tipstextimagepretrainingspatial] Base Patch14 14.8 59.3 6.9 60.7 59.3 Large Patch14 21.5 59.9 8.9 61.3 63.0 Giant Patch14 20.0 65.2 10.1 66.0 65.6 Voyage-3.5-multimodal 53.3 48.7 30.3 Gemini Embedding 2 79.3 67.7 64.4 90.2 92.1 5.1 Generalization to specialized domains To rigorously assess the versatility and multimodal alignment of Gemini Embedding 2 in specialized contexts, we evaluated its zero-shot image-to-text retrieval capabilities across a diverse suite of domain-specific datasets. To ensure a comprehensive evaluation, we selected datasets corresponding to distinct real-world applications: microscopy and bioscience (MicroVQA [burgess2025microvqa]), fine art (ArtCap [lu2022artcap]), astronomy (AstroLLaVA [zaman2025astrollava]), and culinary arts (Recipe1M [marin2021recipe1m+]). Formulated as a standard Recall@5 (R@5) benchmark, we compared our model against an array of open-source and proprietary vision-language models (see Table˜4). Our findings demonstrate that Gemini Embedding 2 achieves state-of-the-art performance across all evaluated domains, frequently establishing substantial margins of improvement over existing baselines. For instance, in astronomy (AstroLLaVA) and microscopy (MicroVQA), Gemini Embedding 2 achieves a R@5 of 64.4 and 79.3, respectively, effectively doubling the performance of these baselines in astronomy, and outperforming them by over 48% in microscopy. On the Recipe1M dataset, it breaks the 90.0 barrier for retrieving both ingredients (90.2) and instructions (92.1), decisively outperforming the next-best model, SigLIP2-Giant (81.2 and 80.4). Beyond absolute performance margins, our evaluation highlights a notable difference in cross-domain consistency. While the performance of existing model families often fluctuates significantly depending on the target domain, Gemini Embedding 2 maintains a robust, general-purpose alignment. As shown in Table˜4, many baseline architectures exhibit incidental performance peaks and valleys across different specialized domains. For instance, the TIPS [maninis2025tipstextimagepretrainingspatial] model family demonstrates strong alignment in the fine art domain, with TIPS-G14 achieving a R@5 of 65.2 on ArtCap. Yet its performance is comparatively much lower on microscopic biological imagery (20.0 on MicroVQA). Similarly, while the SigLIP2 lineage excels at the Recipe1M dataset (scoring up to 81.2), it struggles to capture the visual semantics of ArtCap (dropping to 8.4). Conversely, Gemini Embedding 2 does not exhibit these sharp, domain-dependent fluctuations. Instead, it offers a consistently reliable multimodal embedding space that generalizes predictably across a diverse array of highly specialized tasks. Ultimately, these results underscore the unprecedented robustness of Gemini Embedding 2’s representations out-of-the-box. Users—ranging from bench biologists and astrophysicists to culinary platforms and digital humanities researchers—can readily integrate Gemini Embedding 2 into their diverse workflows to power highly-accurate, domain-aware, multimodal retrieval systems. 5.2 Impact of synthetic data The text-only Gemini Embedding model [lee2025geminiembeddinggeneralizableembeddings] showed the effectiveness of the Gemini model to improve the quality of the text data used to train the Gemini Embedding model. In this new Gemini Embedding 2 model, we also used the power of Gemini to improve the quality of the data used to train the model. We illustrate this with some of the MTEB Code tasks as example of the impact of Gemini when it is used to synthesize high-quality training data. The results are shown in Table˜5. Considering the results of the text-only Gemini Embedding model as baseline, the equivalent results of the multimodal Gemini Embedding 2 model show some improvement, even before adding any synthetic data. This is remarkable because, as it has been observed in other text-only evaluations, the new multimodal model surpasses the performance of our previous text-only version (refer to Table˜2 for an MMTEB comparison). Adding synthetic data generated with Gemini, results in very noticeable improvements in the three MTEB Code tasks subject of this analysis, especially in the CodeFeedbackMT [zheng2024opencodeinterpreterintegratingcodegeneration] task and also in the SyntheticText2SQL and CodeFeedbackST [li2024coircomprehensivebenchmarkcode] ones. Overall, the use of synthetic data gives a remarkable improvement of +15.81 points in average over our previous Gemini Embedding model in these challenging code retrieval tasks. 媒体内容 · 前往原文查看 Table 5: Results on selected MTEB Code v1 tasks using synthetic datasets. Ablation models exclude souping. Average CodeFeedbackMT CodeFeedbackST SyntheticText2SQL Gemini Embedding 70.5 56.3 85.3 70.0 Gemini Embedding 2 w/o Synthetic 73.0 57.9 85.5 75.7 Gemini Embedding 2 w/ Synthetic 86.3 (+15.8) 92.3 88.6 78.1 5.3 Impact of Fine-Tuning and Pre-Fine-Tuning Figure 3: Comparing Pre-Fine-Tuning (PFT) and Fine-Tuning (FT) checkpoints on multimodal evals. We compare the performance of the Pre-Fine-Tuning (PFT) checkpoint and the final Fine-Tuning (FT) checkpoint across various image and video understanding tasks. As shown in Figure˜3, FT improves performance over PFT across almost all evaluated benchmarks. The improvements on image tasks, while consistent, are relatively modest. The most significant improvements are concentrated in the video evaluations due to the additional video training data in FT. 5.4 Impact of In-Domain Video Data 媒体内容 · 前往原文查看 Model Configuration MSR-VTT YouCook2 Vatex nDCG@10 nDCG@10 nDCG@10 Baseline Gemini Embedding 2 68.2 – 55.9 – 69.2 – Fine-Tuned () Models + MSR-VTT data () 75.0 +6.8 56.1 +0.2 71.7 +2.5 + MSR-VTT & Vatex data () 76.1 +7.9 55.3 -0.6 79.5 +10.3 Model Soups (Gemini Embedding 2 : ) Ratio 2:1 () 71.7 +3.5 56.1 +0.2 74.5 +5.3 Ratio 1:1 () 73.7 +5.5 56.8 +0.9 76.8 +7.6 Table 6: Summary of video metrics (NDCG@10 in %) for fine-tuned and souped models. The columns indicate absolute percentage point differences relative to the Gemini Embedding 2 baseline. Adding targeted data improves in-domain performance but can slightly degrade out-of-domain tasks (e.g., YouCook2 dipping by 0.6%), whereas model souping effectively balances these task-specific gains with the original model’s robustness. Comparing the fine-tuned models built on top of Gemini Embedding 2, Table˜6 shows that the evaluation metrics are highly sensitive to the addition of targeted, in-domain data. Note that we add the in-domain data into the finetuning mixture and train one epoch of the added data. With only a few thousand steps of training and modest O(k) data quantities , we can drive significant improvements in targeted tasks (e.g., adding MSR-VTT and Vatex’s training splits pushes MSR-VTT to 76.1% and Vatex to 79.5%). However, this narrow focus can lead to slight degradations in out-of-domain tasks (such as YouCook2 dipping to 55.3%). Interestingly, the newly fine-tuned weights remain highly compatible with the original base model through model souping. Simple interpolation of the souping weights (such as the or mixtures) effectively brings back the video performance gains, in several cases yielding better results across the board than the baseline by balancing task-specific knowledge with the robustness of the original model. 6 Future Work The vast native multimodal capabilities of Gemini Embedding 2 unlocks the potential for numerous enterprise use cases like agentic RAG, video recommendation, interleaved multimodal retrieval, etc. without the need for conversion to intermediate modalities. With LLM backbones being highly capable, we believe including other signals from search systems like ranking can be hugely beneficial to improving the retrieval capabilities of embeddings. Agentic RAG use cases also point towards potential future directions of training end-to-end RAG use cases with embeddings being fine-tuned for these enterprise use cases. As the scope of interleaved multimodal applications continues to expand, we invite the broader academic community to contribute novel evaluation frameworks to help benchmark these emerging capabilities. 7 Conclusion Gemini Embedding 2 represents a transformative step forward in general-purpose representation, delivering a state-of-the-art multimodal successor to our text-only Gemini Embedding model. Gemini Embedding 2 generalizes well across a wide variety of tasks by seamlessly producing embeddings for arbitrary combinations of interleaved inputs across all modalities including text, image, audio, and video. By leveraging Gemini’s core multimodal, multilingual and code-centric foundations, the Gemini Embedding 2 model achieves landmark performance on well-known embedding benchmarks like MSCOCO, Vatex and MMTEB with a particularly significant leap in code retrieval. Our findings highlight its remarkable versatility, showing that it excels not only in general tasks but also across specialized domains such as microscopy, astronomy, and the culinary arts. Furthermore, by demonstrating that native audio input outperforms traditional ASR in retrieval tasks and removing the need for costly task-specific instructions, Gemini Embedding 2 offers a highly efficient architecture. This unified approach to embedding facilitates a sophisticated cross-data retrieval setup, providing the essential infrastructure for building next-generation agentic systems in tandem with Gemini. References 8 Full Results 媒体内容 · 前往原文查看 Table 7: Full results of Gemini Embedding 2 on MTEB (Multilingual). Task Name Performance AILAStatutes 49.50 AfriSentiClassification 59.38 AlloProfClusteringS2S.v2 61.75 AlloprofReranking 84.16 AmazonCounterfactualClassification 86.99 ArXivHierarchicalClusteringP2P 63.86 ArXivHierarchicalClusteringS2S 64.54 ArguAna 83.60 ArmenianParaphrasePC 97.56 BUCC.v2 99.09 BelebeleRetrieval 93.81 BibleNLPBitextMining 34.09 BigPatentClustering.v2 41.59 BiorxivClusteringP2P.v2 53.10 BornholmBitextMining 64.14 BrazilianToxicTweetsClassification 33.21 BulgarianStoreReviewSentimentClassfication 81.32 CEDRClassification 57.13 CLSClusteringP2P.v2 43.56 CSFDSKMovieReviewSentimentClassification 54.92 CTKFactsNLI 87.20 CataloniaTweetClassification 58.76 Core17InstructionRetrieval 6.44 CovidRetrieval 80.14 CyrillicTurkicLangClassification 95.16 CzechProductReviewSentimentClassification 68.47 DBpediaClassification 93.83 DalajClassification 51.26 DiaBlaBitextMining 89.00 EstonianValenceClassification 54.47 FaroeseSTS 88.83 FilipinoShopeeReviewsClassification 50.11 FinParaSTS 32.37 FinancialPhrasebankClassification 87.16 FloresBitextMining 90.43 GermanSTSBenchmark 87.90 GreekLegalCodeClassification 51.65 GujaratiNewsClassification 92.19 HALClusteringS2S.v2 32.30 HagridRetrieval 99.19 IN22GenBitextMining 98.43 IndicCrosslingualSTS 61.36 IndicGenBenchFloresBitextMining 99.22 IndicLangClassification 83.39 IndonesianIdClickbaitClassification 64.95 IsiZuluNewsClassification 46.16 ItaCaseholdClassification 69.68 JSICK 84.85 KorHateSpeechMLClassification 26.39 KorSarcasmClassification 64.39 KurdishSentimentClassification 87.90 LEMBPasskeyRetrieval 62.25 LegalBenchCorporateLobbying 96.37 MIRACLRetrievalHardNegatives 71.15 MLQARetrieval 84.51 MacedonianTweetSentimentClassification 73.13 MalteseNewsClassification 39.76 MasakhaNEWSClassification 82.63 MasakhaNEWSClusteringS2S 60.44 MassiveIntentClassification 80.86 MedrxivClusteringP2P.v2 46.53 MultiEURLEXMultilabelClassification 4.70 MultiHateClassification 79.31 NTREXBitextMining 96.48 NepaliNewsClassification 97.98 News21InstructionRetrieval 2.64 Task Name Performance NollySentiBitextMining 76.46 NordicLangClassification 90.34 NorwegianCourtsBitextMining 95.39 NusaParagraphEmotionClassification 62.17 NusaTranslationBitextMining 84.47 NusaX-senti 85.26 NusaXBitextMining 93.04 OdiaNewsClassification 95.78 OpusparcusPC 97.11 PAC 70.75 PawsXPairClassification 61.22 PlscClusteringP2P.v2 75.65 PoemSentimentClassification 57.27 PolEmo2.0-OUT 77.00 PpcPC 95.40 PunjabiNewsClassification 83.12 RTE3 89.79 Robust04InstructionRetrieval -0.44 RomaniBibleClustering 47.88 RuBQReranking 77.98 SCIDOCS 25.68 SIB200ClusteringS2S 43.33 SICK-R 83.59 SNLHierarchicalClusteringP2P 59.59 STS12 81.07 STS13 89.69 STS14 85.48 STS15 90.67 STS17 88.96 STS22.v2 70.80 STSB 85.02 STSBenchmark 88.68 STSES 76.66 ScalaClassification 54.30 SemRel24STS 74.87 SentimentAnalysisHindi 74.48 SinhalaNewsClassification 82.82 SiswatiNewsClassification 57.63 SlovakMovieReviewSentimentClassification 93.57 SpartQA 8.74 SprintDuplicateQuestions 96.61 StackExchangeClustering.v2 92.18 StackOverflowQA 97.76 StatcanDialogueDatasetRetrieRetrieval 63.11 SwahiliNewsClassification 65.71 SwednClusteringP2P 45.96 SwissJudgementClassification 61.77 T2Reranking 67.72 TERRa 64.52 TRECCOVID 77.57 Tatoeba 89.35 TempReasonL1 7.77 ToxicConversationsClassification 85.85 TswanaNewsClassification 53.92 TweetTopicSingleClassification 73.15 TwitterHjerneRetrieval 94.54 TwitterURLCorpus 88.07 VoyageMMarcoReranking 71.89 WebLINXCandidatesReranking 19.01 WikiCitiesClustering 79.46 WikiClusteringP2P.v2 28.51 WikipediaRerankingMultilingual 93.25 WikipediaRetrievalMultilingual 94.82 WinoGrande 69.57 XNLI 78.95 indonli 59.21 媒体内容 · 前往原文查看 Table 8: Full results of Gemini Embedding 2 on MTEB(Code). Task Name Performance AppsRetrieval 98.60 COIRCodeSearchNetRetrieval 91.90 CodeEditSearchRetrieval 91.94 CodeFeedbackMT 92.30 CodeFeedbackST 88.59 CodeSearchNetCCRetrieval 96.25 CodeSearchNetRetrieval 92.96 CodeTransOceanContest 93.19 CodeTransOceanDL 33.72 CosQA 52.05 StackOverflowQA 97.89 SyntheticText2SQL 78.11 9 Contributions and Acknowledgments Core Contributors (∗: equal contributions) Madhuri Shanbhogue∗ Zhe Li∗ Shanfeng Zhang∗ Gustavo Hernández Ábrego∗ Shih-Cheng Huang∗ Aashi Jain∗ Daniel Salz Sonam Goenka Chaitra Hegde Ji Ma Feiyang Chen Jiaxing Wu Tanmaya Dabral Babak Samari Kevin Poulet Daniel Cer Kaifeng Chen Paul Suganathan Hui Hui Jovan Andonov Philippe Schlattner Jay Han Iftekhar Naim Wing Lowe Vladimir Pchelin Albert Yang Yi-Ting Chen Zhongli Ding Grace Zhang Georg Heigold Yichang Chen Antoine Reveillon Brendan Mccloskey Wenlei Zhou Dahun Kim Rui Meng Emma Wang Jack Zheng Halley Fede Zhen Yang Keegan Mosley Brian Potetz Sahil Dua Henrique Schechter Vera Shen Gao Hesen Zhang Andreas Hess Hengxuan Ying Alberto Montes Karan Gill Min Choi Sebastian Russo Anja Hauth Jinhyuk Lee Michael Boratko Megan Barnes Vikram Rao Claudiu Musat Cyril Allauzen Ehsan Variani Shankar Kumar Tom Bagby Junyi Jiao Yang Gu Tengxin Li Ayush Agrawal Roberto Santana Dev Nath Stephen Karukas Shuoxuan Han Lucia Loher Alice Twu Nidhi Vyas Siddharth Bhai Frank Palma Gomez Wangyuan Zhang Chaoren Liu Jizheng Yang Steve Qiu Shijie Zhang Sujay Kulkarni Sascha Rothe Sean Nakamoto Leadership Raphael Hoffmann Zach Gleicher Yunhsuan Sung Qin Yin Tom Duerig Mojtaba Seyedhosseini Acknowledgement James Gan, Jon Matthews, Luciano Martins, Patrick Löber, Anna Kelly, Kristen Quan, Roxanne Daniel, Ryan Trostle, Tania Bedrax-Weiss, Srinivasan (Cheenu) Venkatachary, Howard Zhou, Tomas Izo.