Abstract

Deep search has become a crucial capability for frontier multimodal agents, enabling models to solve complex questions through active search, evidence verification, and multi-step reasoning. Despite rapid progress, top-tier multimodal search agents remain difficult to reproduce, largely due to the absence of open high-quality training data, transparent trajectory synthesis pipelines, or detailed training recipes. To this end, we introduce OpenSearch-VL, a fully open-source recipe for training frontier multimodal deep search agents with agentic reinforcement learning. First, we curated a dedicated pipeline to construct high-quality training data through Wikipedia path sampling, fuzzy entity rewriting, and source-anchor visual grounding, which jointly reduce shortcuts and one-step retrieval collapse. Based on this pipeline, we curate two training datasets, SearchVL-SFT-36k for SFT and SearchVL-RL-8k for RL. Besides, we design a diverse tool environment that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition. Finally, we propose a multi-turn fatal-aware GRPO training algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping. Built on this recipe, OpenSearch-VL delivers substantial performance gains, with over 10-point average improvements across seven benchmarks, and achieves results comparable to proprietary commercial models on several tasks. We will release all data, code, and models to support open research on multimodal deep search agents.

1 Introduction

Multimodal deep search has emerged as a critical direction for multimodal large language models (MLLMs), enabling them to evolve from passive visual understanding systems into agents that actively search evidence, verify facts, and reason over knowledge-intensive visual queries (Huang et al., 2026; Feng et al., 2026; Chen et al., 2026).However, frontier multimodal search agents remain difficult to reproduce, as their training data, code are often proprietary or insufficiently disclosed (Seed, 2026; Huang et al., 2026; Singh et al., 2025; Team, 2026b). As a result, the community still lacks a fully open recipe for building, analyzing, and improving strong multimodal search agents.Among these missing components, high-quality training data is a central bottleneck. The strongest frontier systems are still largely dominated by well-funded commercial corporations (Team, 2025b; Comanici et al., 2025), where the data sources, filtering criteria, expert demonstrations, and tool-use trajectories are typically kept private.This makes it difficult to reproduce advanced multimodal search capabilities or systematically study which data properties are essential for agentic search behavior.The issue is even more pronounced in multimodal settings, where effective training data must capture image-grounded understanding, multi-hop retrieval, evidence verification, and long-horizon tool use rather than simple visual question answering.Therefore, releasing high-quality training data is crucial for making frontier multimodal search agent research more transparent, reproducible, and accessible.Beyond data, training multimodal search agents also poses unique challenges, especially when applying agentic reinforcement learning (agentic RL) (Fan et al., 2026; Geng et al., 2025; Huang et al., 2026) to long-horizon tool-use settings.Agentic search trajectories involve multiple rounds of reasoning, tool invocation, and observation integration, where a single malformed call, timeout, irrelevant query, or repeated failure can invalidate the remaining rollout.Simply discarding such trajectories wastes useful pre-failure reasoning, while training on the full rollout introduces noisy gradients from meaningless post-failure tokens.Another practical challenge is that real-world visual inputs are often imperfect, such as blurred photos, low-resolution thumbnails, skewed documents, and crowded screenshots.In these cases, searching alone is insufficient, and the agent must first crop, enhance, rectify, or parse the visual evidence before reliable search can begin.However, most existing multimodal search agents focus mainly on retrieval and do not jointly address robust visual pre-processing and failure-aware long-horizon RL.In this work, we introduce OpenSearch-VL, a fully open recipe for training frontier multimodal deep search agents with agentic RL.Our recipe addresses the above challenges from data, tools, and training.First, we develop a dedicated data curation pipeline to build high-quality training data.Starting from the Wikipedia hyperlink graph, we sample multi-hop entity paths and convert them into multi-hop VQA instances by rewriting intermediate entities into fuzzy descriptions, followed by a carefully designed filtering mechanism.This design avoids single-hop image lookup shortcuts and encourages the agent to learn multi-hop search and reasoning behaviors.This pipeline yields two training datasets SearchVL-SFT-36k for SFT and SearchVL-RL-8k for agentic RL.Second, we build a tool environment that goes beyond retrieval-only multimodal agent.In addition to search, the agent is equipped with OCR, cropping, sharpening, super-resolution, and perspective correction, allowing it to handle imperfect visual inputs in real-world scenarios before querying external knowledge.Finally, we develop an agentic RL algorithm based on GRPO (Guo et al., 2025) for long-horizon multimodal tool use, where multi-step interactions often lead to cascading tool failures. To address this issue, we introduce fatal-aware token masking that removes invalid post-failure suffixes from optimization, while preserving useful pre-failure reasoning through one-sided advantage clamping. This enables the model to learn from partially successful trajectories without being affected by noisy gradients from failed rollouts.Together, these designs enable OpenSearch-VL to learn robust long-horizon search behavior over multimodal evidence in real-world scenarios.Experiments across multimodal deep search benchmarks show that OpenSearch-VL consistently improves over strong baselines.For example, compared with the Qwen3-VL-30B-A3B (Bai et al., 2025) agentic baseline, our model improves the average score from 47.8 to 61.6, with large gains on VDR (+13.3) (Zeng et al., 2026), MMSearch (+24.5) (Jiang et al., ), FVQA (+10.2) (Wang et al., 2017), and InfoSeek (+16.2) (Chen et al., 2023).Moreover, OpenSearch-VL achieves comparable or even better performance than proprietary commercial models on several benchmarks.In summary, our main contributions can be summarized as follows:

• We introduce OpenSearch-VL, a fully open recipe for training frontier multimodal deep search agents.We will release the training data, code, and models to provide an open foundation for reproducible research on multimodal agentic search.

• We build the key components required for training advanced multimodal search agents, including high-quality image-grounded multi-hop training data, a diverse tool environment, and a multi-turn fatal-aware GRPO algorithm.

• Extensive experiments demonstrate the effectiveness of our recipe. For example, our trained OpenSearch-VL-30B-A3B brings an average improvement of 13.8 points across 7 multimodal deep search benchmarks.

2 Preliminaries

Problem Formulation.Given an input image and a question , the agent answers by interleaving reasoning with tool calls over a diverse tool set , where contains visual tools that transform or parse images and contains retrieval tools that query external knowledge.At step , the model conditions on the accumulated history

(1)

where , , and denote the images, actions, and observations accumulated up to step .The interaction unfolds as a multi-turn trajectory

(2)

where the final step emits the answer without a subsequent observation. Following the ReAct (Yao et al., 2022) think-then-act convention, each action decomposes as , where is a reasoning trace, and denotes a tool invocation for or the final response for .Multimodal Observations and Active Visual Context.Unlike text-only formulations (Jin et al., 2025), our environment returns multimodal observations. Given a control command , deterministically routes the invocation by tool family,

(3)

so that . The active visual context grows monotonically as ; historical visual observations are strictly preserved so that the policy can cross-reference multi-hop visual transformations (e.g. a localised Crop against its SuperResolution-enhanced counterpart). The rollout is compactly written as , where denotes the strict interleaving of policy-emitted actions and environment-returned observations.Trajectory Likelihood.The policy models the joint trajectory probability via standard autoregressive factorisation:

(4)

Observations are excluded from the generative probability mass since they are exogenous outputs of ; they influence the trajectory likelihood only by modulating subsequent histories for . This factorisation is the object directly supervised by SFT (Eq. 8) and the basis of the per-token importance ratio in our RL objective (Eq. 12).Token-level Generation Mask.Optimisation gradients must be restricted to tokens emitted by the policy itself. For textual observations (originating from and OCR), we define an indicator with iff token is constituent to a generated action , and if belongs to an observation span . Image-valued observations (from ) are injected directly into the visual backbone and inherently bypass the token-level loss. This protocol, inspired by the retrieved-token masking of Search-R1 (Jin et al., 2025), underlies both the SFT objective (Eq. 8) and the fatal-aware RL mask (Eq. 10); textual serialisations of search results and OCR parses are characteristically noisy and structurally divergent from the policy’s intrinsic generative distribution, and including them in the loss destabilises training.Search Tools.OpenSearch-VL is equipped with a suite of tools covering three complementary functions: retrieval (TextSearch, ImageSearch) for gathering external evidence, image enhancement (Sharpen, SuperResolution, PerspectiveCorrect) for remedying low-quality inputs, and attention and parsing (Crop, OCR) for localizing and decoding fine-grained content. The suite combines lightweight offline primitives with online services backed by expert models, and is summarized in Table 1. Full specifications are deferred to Appendix F.

媒体内容 · 前往原文查看

Tool	Description	Arguments	Tool Output
TextSearch	Web search with page reading and LLM summarization	Query + TopK	Query-focused passage summaries
ImageSearch	Reverse image / visual entity search over the web	Image + TopK	Visual matches and related webpages
Sharpen	Unsharp-masking based deblurring / detail enhancement	Image + Amount	Sharpened image
SuperResolution	Deep super-resolution (EDSR) for low-resolution inputs	Image + Scale	High-resolution image
PerspectiveCorrect	Auto perspective rectification of skewed documents	Image	Fronto-parallel image
Crop	Extract a user-specified rectangular region	Image + Coordinates	Cropped image
OCR	Structured document parsing with text and layout labels	Image + Flags	Text blocks with labels and reading order

3 Dataset Curation

To equip the model with robust reasoning and tool-use capabilities, we design a scalable data curation pipeline (Figure 1) that synthesizes high-quality trajectories without manual human annotation. The pipeline proceeds in three stages—VQA construction, staged filtering and enhancement, and trajectory synthesis—yielding the final dataset used for the following stage training.

3.1 High-Quality VQA Construction

A central challenge for training multimodal search agents is the supply of questions that encourage non-trivial use of the diverse tool set . Directly prompting a VLM on an image tends to yield shallow, perception-level queries that can be resolved in a single forward pass (Geng et al., 2025; Huang et al., 2026).Building on this observation, we adopt a unified construction pipeline: we sample multi-hop trajectories over the Wikipedia hyperlink graph, synthesize textual QA pairs along each trajectory, and lift them into image-grounded VQA via answer-preserving fuzzy rewriting and source-anchored visual grounding. Compared with prior QA constructions (Wu et al., 2025a; Li et al., 2025a; Geng et al., 2025), our pipeline (i) assigns each node on the sampled path an explicit functional role within the reasoning chain, and (ii) deliberately decouples the visual anchor from the answer entity, thereby suppressing single-shot retrieval shortcuts.Wikipedia Path Sampling.We cast the Wikipedia (48) as a directed graph with articles as nodes and in-article hyperlinks as edges. From a seed , a constrained random walk of length produces a path

(5)

where each relation is induced by the hyperlink’s anchor text. The walk skips (i) disambiguation and list pages, (ii) cycles, and (iii) hub nodes whose in-degree exceeds a threshold ; full thresholds, resampling heuristics, and additional filters are deferred to Appendix D.2. Each node on is assigned a functional role: is the anchor (visual entry point, to be replaced by a visual referring expression), are bridge nodes (intermediate entities with fuzzified names), and is the answer node (source of the target attribute). These roles govern the rewriting and grounding stages below.We extract a short, unambiguous answer from and prompt GPT-4o (Team, 2024) to synthesize a canonical question that verbalizes and references only through the queried attribute (extraction details in Appendix D.2). The canonical is not a training target but a manipulable object for rewriting.Fuzzy Entity Rewriting.Preserving entity names in enables the agent to short-circuit the chain with a single retrieval (Li et al., 2025a; Huang et al., 2026). We therefore progressively rewrite into a fuzzy counterpart while fixing . Following the iterative style of Skywork-R1V4 (Zhang et al., 2025), we rewrite one entity at a time, from the farthest bridge toward : each name is replaced by a relational or attribute-based descriptor drawn from the entity’s Wikipedia context, and an LLM uniqueness evaluator verifies that the substitution still resolves to the intended entity conditional on the partially rewritten question. A rewrite is accepted only when

(6)

where denotes the set of entities compatible with under the evaluator’s world knowledge. We further interleave entity rewriting with occasional answer obfuscation (Huang et al., 2026) to avoid collapsing onto a stereotyped relational template.Anchor-aware Visual Grounding.We retrieve a representative image of the anchor from Wikimedia Commons or its Wikipedia infobox, filter candidates by CLIP similarity to a short textual description of , and replace in with a visual referring expression (e.g., “the person in the image”) to yield the final question . Unlike prior QA-to-VQA conversions (Geng et al., 2025; Zhang et al., 2025) that ground on or near the answer entity, anchoring at the source of substantially reduces single-hop shortcuts: the agent must first identify the visual anchor and then follow the intermediate textual relations before reaching .Each candidate triple is gated by automatic checks for masking, uniqueness, and visual relevance, generalizing the selector/examiner protocol of WebWatcher (Geng et al., 2025) (full criteria in Appendix D.2); non-triviality is handled jointly with the staged filtering of Sec. 3.2. Instances passing these checks form the Wikipedia portion of our VQA pool, subsequently merged with open-source multimodal corpora before trajectory synthesis.

3.2 Filtering and Enhancement

Before trajectory synthesis, we consolidate the Wikipedia-derived VQA instances from Sec. 3.1 with three open-source multimodal corpora—LiveVQA (Fu et al., 2025), FVQA (Wang et al., 2017), and WebQA (Chang et al., 2022)—to broaden coverage across live entities, commonsense fact lookup, and open-web multi-hop reasoning. We then apply a two-stage difficulty filter using a frozen Qwen3-VL-32B (Bai et al., 2025): first discarding examples answerable without tools, and then discarding examples solvable with a single ImageSearch call. This removes samples that rely only on parametric knowledge, perceptual shortcuts, answer-coincident anchors, or one-hop bridge leakage, ensuring that retained instances genuinely require the intended visual-to-text search chain.To further expose the agent to realistic visual imperfections, we randomly select of the filtered VQA pool and apply controlled degradations—blur, downsampling, and perspective distortion—paired with the corresponding enhancement tools in (Sharpen, SuperResolution, and PerspectiveCorrect). This enhancement subset diversifies the training distribution and induces a think-with-image behavior: when the input image is unreliable, the policy learns to repair the visual evidence before initiating retrieval. Together, the filtered retrieval-heavy instances and the enhancement-required subset exercise both visual restoration and evidence acquisition within the unified tool environment.

3.3 Multi-turn Trajectory Synthesis

For each instance that survives the filters of Sec. 3.2, we synthesize expert trajectories by rolling out Claude Opus 4.6 (Team, 2026a) as the expert model against the real execution environment , prompted with the agent system prompt of Appendix E and free to invoke any tool in . We draw independent rollouts per instance, each formatted as a multi-turn ReAct (Yao et al., 2022) trajectory aligned with Eq. 2. Then the raw rollouts are passed through a two-stage rejection cascade. The first stage discards any trajectory whose final answer disagrees with the ground truth (adjudicated by the same GPT-4o (Team, 2024) LLM-as-judge (Gu et al., 2025) we use for , Sec. 4.2). The surviving trajectories are then vetted by a GPT-5.4 process-level judge on tool-use, logical consistency between reasoning and observations, and absence of ineffective repetition, sharing the four-dimension rubric of (Sec. 4.2).Applying both stages to the full rollout corpus yields high-quality expert trajectories with an average of tool-invocation turns per trajectory, which together constitute the SFT corpus consumed in Sec. 4.1.

4 Training

We train OpenSearch-VL in two sequential stages. First, we perform supervised fine-tuning (SFT) to instill fundamental reasoning and tool-use behaviors; subsequently, we apply reinforcement learning (RL) via a multi-turn, search-augmented objective (Figure 2) to discover more effective exploration strategies.

4.1 Supervised Fine-Tuning

We perform SFT on a curated set of multi-turnexpert trajectories (Section 3).Using the history (Eq. 1) and the actiondecomposition , by autoregressive factorisation thestep-level action probability decomposes as

(7)

Summing over all trajectories and steps, the standard SFT objective can be equivalentlywritten as

(8)

where tool observations enter only as conditioning context and areexcluded from the loss computation following the retrieved-token maskingstrategy of (Jin et al., 2025).This provides a structured interpretation of the training signal,showing that it jointly supervises both the reasoning trace and thesubsequent tool invocation (or terminal response) at each step.

4.2 Multi-Turn Search Fatal-Aware GRPO

While SFT provides a strong initialization for tool use, it remains bounded by the coverage of the demonstration trajectories and therefore cannot discover improved search strategies through exploration. We address this limitation with reinforcement learning, building on GRPO (Shao et al., 2024) and its search-augmented extension (Jin et al., 2025). Our setting, however, differs from prior search-only formulations in three important respects: we optimize over a multimodal environment with diverse tools rather than a text-only retriever ; we use a composite reward that combines final-task success with process-level search quality; and we introduce fatal-aware masking together with one-sided advantage clamping to preserve useful supervision from partially successful trajectories.During training, for each prompt we sample a group of multi-turn rollouts , for .Composite Multi-Turn Reward.Long-horizon tasks pose a sparse-reward challenge: outcome-only rewards miss credit for partially successful reasoning, while process-only rewards risk misalignment from the end goal. We therefore use a composite trajectory-level reward

(9)

where . The composite trajectory-level reward (Eq. 9) is structured to balance algorithmic formatting, terminal accuracy, and process-level search quality. We define each component as follows:

• Format reward . A deterministic, algorithmic prior that enforces structural integrity. We define , where iff step emits a contiguous <think></think> block immediately followed by either a <tool_call></tool_call> block (for ) or a <response></response> block (for ); for any structural violation, including steps that trigger tool-execution errors (e.g., malformed API arguments). By acting as a multiplicative gate in Eq. 9, drives the overall return of structurally degraded trajectories toward zero.

• Accuracy reward . A terminal outcome metric assessing the fidelity of the agent’s final resolution. A GPT-4o (Team, 2024) judge under an LLM-as-Judge protocol (Gu et al., 2025) verifies semantic equivalence between the agent’s terminal <response> and the ground-truth annotation, assigning for a match and otherwise. Convention for truncated trajectories: for trajectories aborted by the fatal-state condition (Eq. 10) before emitting a terminal <response>, we deterministically assign . This is a structural guarantee rather than an evaluative judgment—in the absence of a terminal answer, correctness is strictly undefined and conservatively zeroed out—and it ensures the outcome signal remains well-defined for group-relative advantage estimation regardless of completion status.

• Query-quality reward . A process-level signal that counteracts the inherent sparsity of in long-horizon interactions. We use GPT-5.4 (OpenAI, 2025), a proprietary frontier reasoning model, as the query-quality judge to score the cumulative sequence of search queries on a continuous scale, providing dense feedback for unsuccessful trajectories with . The rubric covers four dimensions: (i) semantic relevance of issued queries to the initial prompt; (ii) logical progression and iterative refinement of queries across successive turns; (iii) signal-to-noise ratio within retrieved payloads; and (iv) cross-modal complementary use of image and text retrieval tools. For trajectories designated as fatal, the judge restricts its evaluation to the valid pre-fatal prefix (steps ), so that early-stage reasoning is credited despite subsequent collapse.

Fatal-Aware Token Masking.In unconstrained multi-turn environments, agents frequently encounter “fatal” states—such as cascading tool-execution failures or infinite loops—after which subsequent reasoning becomes meaningless. Standard approaches either discard the entire trajectory (Huang et al., 2026) (wasting the valid early steps) or train on it blindly (injecting noise). We introduce a fatal-aware masking strategy to preserve the viable prefix.We define the fatal step index for trajectory as the earliest step where consecutive tool-execution errors commence, with if no such cascade occurs. We then extend the observation-token (generation) mask to additionally zero out all tokens generated after the fatal step:

(10)

where maps token index to its step index .Crucially, the process rewards and are computed exclusively over the valid prefix , ensuring the model is not penalised for structural collapse that occurs after the trajectory is deemed fatal.All trajectories—including fatal ones—contribute to the standard group-normalised reward to keep group statistics unbiased. However, directly using for fatal trajectories is pathological: a sub-mean would push the policy gradient to suppress the viable prefix, discouraging the valid reasoning before the error cascade. We therefore apply one-sided advantage clamping:

(11)

This clamping ensures that the valid prefix of a fatal trajectory is only ever reinforced if its partial reward exceeds the group mean, and otherwise receives zero gradient rather than an undeserved penalty. In this sense, it generalizes the hard-masking baseline (Huang et al., 2026) while recovering strictly more useful learning signal.Integrating the fatal-aware mask and the clamped advantage , our final GRPO objective over the multimodal environment is formulated as:

(12)

where is the token-level importance ratio, and the standard term is omitted from the display since it is identical to that of standard GRPO (Eq. 14). Relative to search-augmented GRPO (Jin et al., 2025), the differences (color-matched to Eq. 12) are threefold: the execution environment is generalized from to , the generation mask is extended to the fatal-aware mask , and the advantage is computed from the composite reward with one-sided clamping. Additional details and derivations are provided in Appendix B.

5 Experiments

We first describe our experimental setups as follows:Models and Benchmarks.Our OpenSearch-VL is built on three Qwen3-VL variants (Bai et al., 2025): Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct, and Qwen3-VL-32B-Instruct. For evaluation, we use seven knowledge-intensive benchmarks from our main results: SimpleVQA (Cheng et al., 2025), VDR (Zeng et al., 2026), MMSearch (Jiang et al., ), LiveVQA (Fu et al., 2025), BrowseComp-VL (Geng et al., 2025), FVQA (Wang et al., 2017), and InfoSeek (Chen et al., 2023). Together, they cover visual entity recognition, web evidence retrieval, multi-hop reasoning, and long-tail QA.Baselines and Evaluation Metrics.We evaluate OpenSearch-VL against baseline types: Direct Reasoning, where the model answers from its parametric knowledge and visual perception alone; RAG Workflow, where external retrieval results are provided in-context but the reasoning remains single-pass; and Agentic Workflow, where the model autonomously interleaves reasoning with tool calls in a multimodal environment. We report Pass@1 on all seven benchmarks. Correctness is adjudicated by a GPT-4o judge that compares the model’s final response with the reference answer. To ensure fair comparison across heterogeneous answer styles, we adopt the same evaluation protocol as VDR-Bench (Zeng et al., 2026); the full judge prompt is provided in Figure 10.Datasets.Our SFT data consists of the 36K multi-turn trajectories synthesized by the procedure in Sec. 3.3. For RL training, we randomly sample 8K examples from the VQA pool after the staged filtering and enhancement process in Sec. 3.2, ensuring that these examples are disjoint from the VQA instances used to synthesize the SFT trajectories.Implementation Details.OpenSearch-VL extends LlamaFactory (Zheng et al., 2024) for agentic SFT and builds on rLLM (Tan et al., 2025) and VDR (Huang et al., 2026) for multi-turn tool-interleaved RL. All stages of OpenSearch-VL are trained on Nvidia H20 GPUs. Agentic SFT takes roughly 2 days for the 8B dense model and 4 days for the 30B-A3B MoE model on 256 H20s (32 nodes 8 GPUs); the subsequent multi-turn fatal-aware GRPO stage runs for approximately 200 optimization steps over 10 days on 64 H20s (8 nodes 8 GPUs). The complete per-stage hyperparameter configurations are listed in the following subsections (SFT in Table 4; RL in Table 5). More details are reported in Appendix C.

5.1 Main Results

媒体内容 · 前往原文查看

Model	SimpleVQA	VDR	MMSearch	LiveVQA	BrowseComp-VL	FVQA	InfoSeek	Avg
Direct Reasoning
GPT-4o (Team, 2024)	51.7	1.7	18.7	28.1	5.5	48.0	52.9	29.5
GPT-5 (OpenAI, 2025)	61.6	9.8	35.1	44.4	48.6	54.4	61.7	45.1
Gemini-2.5-Flash (Comanici et al., 2025)	57.9	6.2	30.4	51.0	37.1	47.7	44.1	39.2
Gemini-2.5-Pro (Comanici et al., 2025)	63.0	8.0	39.8	60.3	43.1	60.7	46.9	46.0
Claude-4-Sonnet (Team, 2025b)	50.9	2.0	18.7	38.5	29.3	35.3	57.3	33.1
Claude-3.7-Sonnet (Team, 2025a)	42.7	4.6	21.1	38.0	32.3	36.7	54.8	32.9
Qwen3-VL-8B (Bai et al., 2025)	47.1	2.8	11.7	23.1	24.1	24.2	23.1	22.3
Qwen3-VL-30B-A3B (Bai et al., 2025)	53.2	3.8	18.7	42.7	29.6	34.7	26.4	29.9
Qwen3-VL-32B (Bai et al., 2025)	58.0	4.1	19.8	45.5	30.8	34.1	28.8	31.6
RAG Workflow
GPT-4o (Team, 2024)	63.6	4.5	49.1	40.1	13.4	66.3	59.5	42.4
GPT-5 (OpenAI, 2025)	55.9	22.3	52.6	56.0	54.9	62.6	70.6	53.6
Claude-3.7-Sonnet (Team, 2025a)	59.3	11.3	32.7	30.3	10.0	59.1	60.2	37.6
Qwen3-VL-8B (Bai et al., 2025)	62.3	7.3	47.3	39.3	29.3	53.6	46.1	40.7
Agentic Workflow
DeepMMSearch-R1-7B (Narayan et al., 2025b)	55.8	–	–	–	–	–	47.5	–
Visual-ARFT-7B (Liu et al., 2025)	42.4	3.3	34.5	25.4	16.5	41.7	37.9	28.8
MMSearch-R1-7B(Wu et al., 2025b)	57.4	2.9	53.8	48.4	20.9	58.4	55.1	42.4
DeepEyes-v2-7B (Hong et al., 2025)	59.4	7.8	63.7	–	–	60.6	51.1	–
WebWatcher-7B (Geng et al., 2025)	54.3	10.3	49.1	51.2	21.2	–	–	–
Qwen3-VL-8B (Bai et al., 2025)	52.0	17.0	37.4	50.6	27.9	58.7	50.3	42.0
SenseNova-MARS-8B (Chng et al., 2025)	61.7	19.4	67.4	56.2	35.1	67.1	61.7	52.7
OpenSearch-VL-8B(Ours)	71.6	20.8	64.5	59.6	37.6	71.5	70.2	56.6
Qwen3-VL-30B-A3B (Bai et al., 2025)	55.1	20.2	44.2	62.0	34.1	63.0	56.2	47.8
OpenSearch-VL-30B-A3B(Ours)	74.9	33.5	68.7	67.4	41.1	73.2	72.4	61.6
Qwen3-VL-32B (Bai et al., 2025)	58.7	23.1	53.9	45.5	35.1	61.2	58.5	48.0
WebWatcher-32B Geng et al. (2025)	59.0	–	55.3	58.7	27.0	–	–	–
OpenSearch-VL-32B(Ours)	76.2	33.8	72.3	70.5	43.8	74.7	74.8	63.7

Table 2 reports the results on seven multimodal knowledge-intensive QA and web-search benchmarks. OpenSearch-VL exhibits a clear advantage over both direct-reasoning and RAG baselines across all scales, underscoring the necessity of an agentic loop for complex multimodal queries. Among 8B-scale agents, OpenSearch-VL-8B achieves the best average score of 56.6, surpassing the previous strongest open 8B agent, SenseNova-MARS-8B, by 3.9 points on average. At larger scales, OpenSearch-VL-30B-A3B and OpenSearch-VL-32B further improve the average score to 61.6 and 63.7, respectively; notably, OpenSearch-VL-32B outperforms strong proprietary direct-reasoning models such as Gemini-2.5-Pro and substantially exceeds the corresponding Qwen3-VL agentic baselines. These results demonstrate that our training recipe scales effectively from 8B to 32B and yields strong gains on both search-heavy and visually grounded benchmarks.

5.2 Ablation Study

We conduct ablations on the Qwen3-VL-8B model to validate the two design choices that define OpenSearch-VL: the data synthesis pipeline that produces tool-demanding multimodal trajectories, and the fatal-aware RL recipe that improves the policy beyond offline imitation.

媒体内容 · 前往原文查看

(a) SFT data pipeline ablation
Settings SimpleVQA InfoSeek FVQA Avg. Full pipeline (Ours) 66.1 62.4 65.3 64.6 w/o source-anchor grounding 53.6(-12.5) 54.5(-7.9) 51.2(-14.1) 53.1(-11.5) w/o fuzzy entity rewriting 51.7(-14.4) 56.4(-6.0) 54.7(-10.6) 54.3(-10.3) w/o staged filtering 57.6(-8.5) 55.2(-7.2) 56.3(-9.0) 56.4(-8.2) w/o enhancement subset 64.9(-1.2) 61.7(-0.7) 63.2(-2.1) 63.3(-1.3)

(b) RL recipe ablation
Method SimpleVQA InfoSeek FVQA Avg. Qwen3-VL-8B 52.0 50.3 58.7 53.7 Qwen3-VL-8B + SFT only 66.1 62.4 65.3 64.6 + Vanilla GRPO 68.8 66.5 67.4 67.6 + GRPO w/ Hard Masking 68.3(-0.5) 67.9(+1.4) 66.9(-0.5) 67.7(+0.1) + GRPO w/ Fatal Masking only 69.7(+0.9) 68.3(+1.8) 69.2(+1.8) 69.1(+1.5) + Fatal Masking + One-sided Clamp (Ours) 71.6(+2.8) 72.4(+5.9) 71.5(+4.1) 71.8(+4.2)

Data Pipeline Ablation.Table 3 shows that each stage of our data synthesis pipeline contributes to the final performance. The full pipeline achieves the best average score of 64.6, while removing source-anchor grounding, fuzzy entity rewriting, or staged filtering leads to large drops of 11.5, 10.3, and 8.2 points, respectively. These results indicate that effective training data must both prevent shortcut retrieval and preserve genuinely tool-demanding queries. Removing the enhancement subset causes a smaller but consistent decline ( Avg.), suggesting that image-restoration trajectories mainly improve robustness rather than driving the core gains.Training Recipe Ablation.Table 3 studies the effect of our RL recipe after the same Qwen3-VL-8B SFT initialization, using Search-R1-style vanilla GRPO (Jin et al., 2025) as the RL baseline. SFT improves the base model from 53.7 to 64.6 average accuracy, and vanilla GRPO further raises it to 67.6, showing the benefit of online exploration. The way fatal trajectories are handled is crucial: the hard-masking strategy of Vision-DeepResearch (Huang et al., 2026) brings almost no gain over vanilla GRPO (Jin et al., 2025; Guo et al., 2025)(), while fatal masking improves the average to 69.1 by preserving valid pre-failure reasoning. Our full method with one-sided advantage clamping achieves the best score on every benchmark and reaches 71.8 average accuracy, a 4.2-point gain over vanilla GRPO. The training curves in Fig. 3 are consistent with this result: fatal-aware GRPO sustains longer tool-use trajectories while achieving higher batch accuracy, indicating that it encourages productive exploration rather than prematurely suppressing difficult rollouts.

Empirical Visualization of the Two Cases.

Figure 4 visualizes how interacts with per-group difficulty on two representative rollout groups. In the higher-difficulty case, most fatal trajectories still emit a coherent prefix before hitting a tool error; a fraction of them beat the group mean and contribute a positive gradient to the prefix tokens only. In the lower-difficulty case, every fatal trajectory is dominated by the successful non-fatal rollouts, and clamping to zero prevents a noisy negative signal from pushing the policy away from what may actually be a valid prefix.Figure 5 shows that this split is not anecdotal: aggregated over groups, the vast majority of fatal trajectories sit on the negative side of the pre-clamp score and are safely zeroed out, while the small preserved tail is distributionally close to the right mode of the non-fatal reference. One-sided clamping thus recovers a principled credit signal from failed tool-call trajectories without amplifying their inherent noise.

6 Related Work

6.1 Multimodal Agentic Search

The integration of active search has reframed LLMs from static knowledge bases into agentic reasoners capable of dynamic information retrieval. Search-R1 (Jin et al., 2025) crystallized this shift by using RL to incentivize autonomous, multi-turn querying within the reasoning chain. This paradigm has since migrated to multimodal domains; MMSearch-R1 (Wu et al., 2025b) and Vision-DeepResearch (Huang et al., 2026; Narayan et al., 2025a) embed visual retrieval into agent pipelines, while recent efforts (Zhang et al., 2025; Yao et al., 2026; Chen et al., 2026; Feng et al., 2026; Li et al., 2025b) attempt to unify image manipulation with web search. However, a common but fragile assumption in these works is the availability of pristine visual inputs. In practice, when agents encounter degraded or text-dense real-world images, the "search-only" approach fails as retrieval cannot fix fundamentally broken visual evidence.

6.2 Visual Perception and Retrieval

The performance of multimodal agents is often bottlenecked not by retrieval logic, but by the fidelity of initial perception Zhang et al. (2026); Wei et al. (2025). While RAG frameworks like VisRAG (Yu et al., 2024) emphasize preserving visual structure, they treat the model as a passive observer that must "make do" with whatever is retrieved. Even as tool-augmented agents (Song et al., 2026; Hong et al., 2025; Geng et al., 2025) introduce dynamic orchestration, their toolsets remain largely homogeneous and scenario-bound. We argue that robust reasoning requires active perception: the agent must not only search but also intervene—autonomously invoking tools like super-resolution or specialized OCR to remediate visual noise before attempting to reason over it.

6.3 Reinforcement Learning for Agentic Reasoning

Training agents to operate in long-horizon, multi-tool environments poses substantial challenges for standard RL. While GRPO (Shao et al., 2024) has shown strong effectiveness in aligning reasoning trajectories for language models (Guo et al., 2025; Chen et al., 2025a, b; Feng et al., 2025a, b; Hu et al., 2026), applying it to agentic rollouts with diverse tool interactions remains non-trivial. The primary challenge lies in cascading failures: an early tool error renders the rest of the trajectory incoherent (Vuddanti et al., 2025; Zhu et al., 2025; Zheng et al., 2026; Dong et al., 2025), yet these post-failure tokens still inject noise into the policy gradient (Liu and Xiao, 2025; Deng et al., 2026; Garg et al., 2026). Furthermore, standard group-normalization tends to penalize the valid reasoning prefixes of partially successful rollouts (Huang et al., 2026). To counter this, we introduce a fatal-aware RL objective that prunes learning signals from post-failure states and uses one-sided advantage clamping to protect the gradients of constructive reasoning steps.

7 Conclusion

We present OpenSearch-VL, a fully open recipe for training multimodal deep search agents with agentic reinforcement learning. Our recipe combines a Wikipedia-based data curation pipeline that mitigates one-step retrieval shortcuts and produces two high-quality datasets: SearchVL-SFT-36k & SearchVL-RL-8k; a diverse tool environment spanning retrieval, image enhancement, and attention-and-parsing tools; and a multi-turn fatal-aware GRPO algorithm that preserves useful pre-failure reasoning through one-sided advantage clamping. Based on this recipe, OpenSearch-VL achieves over 10-point average gains across seven multimodal deep search benchmarks, with competitive performance on representative tasks such as VDR compared with strong proprietary reasoning models. We will release our data, code, models, and training recipe, with the aim to lower the reproducibility barrier and provide an open foundation for future research on multimodal deep search agents.

Limitations and Future Work

A non-trivial fraction of training instability traces to the external tool environment —including search ranking drift, fetch failures, and occasional summarization hallucinations in TextSearch and ImageSearch—which inflates reward variance and motivates future work on on-policy reliability estimation. Furthermore, our composite reward (Eq. 9) relies on proprietary GPT-4o judges, which are costly, version-dependent, and currently score only textual queries while ignoring intermediate visual operations (e.g., Crop); replacing these with open process reward models covering the full visual action space remains a natural next step. Finally, exact numerical reproducibility is challenged by the reliance on these externally hosted APIs (e.g., Serper, PaddleX OCR) and the prohibitive cost of reporting multi-seed error bars for large-scale evaluations (Table 2). To mitigate these constraints and support open research, we will release our complete datasets (SearchVL-SFT-36k / SearchVL-RL-8k), model checkpoints, and training code under permissive licenses.

References

• S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §3.2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, Table 2, §5.
• Y. Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y. Bisk (2022) WebQA: multihop and multimodal qa. External Links: 2109.00590, Link Cited by: §3.2.
• S. Chen, Y. Guo, Z. Su, Y. Li, Y. Wu, J. Chen, J. Chen, W. Wang, X. Qu, and Y. Cheng (2025a) Advancing multimodal reasoning: from optimized cold start to staged reinforcement learning. arXiv preprint arXiv:2506.04207. Cited by: §6.3.
• S. Chen, Y. Guo, Y. Ye, S. Huang, W. Hu, H. Li, M. Zhang, J. Chen, S. Guo, and N. Peng (2025b) ARES: multimodal adaptive reasoning via difficulty-aware token-level entropy shaping. arXiv preprint arXiv:2510.08457. Cited by: §6.3.
• S. Chen, Q. Shou, H. Chen, Y. Zhou, K. Feng, W. Hu, Y. Zhang, Y. Lin, W. Huang, M. Song, et al. (2026) Unify-agent: a unified multimodal agent for world-grounded image synthesis. arXiv preprint arXiv:2603.29620. Cited by: §1, §6.1.
• Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M. Chang (2023) Can pre-trained vision and language models answer visual information-seeking questions?. arXiv preprint arXiv:2302.11713. Cited by: §1, §5.
• X. Cheng, W. Zhang, S. Zhang, J. Yang, X. Guan, X. Wu, X. Li, G. Zhang, J. Liu, Y. Mai, et al. (2025) Simplevqa: multimodal factuality evaluation for multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4637–4646. Cited by: §5.
• Y. X. Chng, T. Hu, W. Tong, X. Li, J. Chen, H. Yu, J. Lu, H. Guo, H. Deng, C. Xie, et al. (2025) SenseNova-mars: empowering multimodal agentic reasoning and search via reinforcement learning. arXiv preprint arXiv:2512.24330. Cited by: Table 2.
• G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1, Table 2, Table 2.
• W. Deng, Y. Li, B. Gong, Y. Ren, C. Thrampoulidis, and X. Li (2026) On group relative policy optimization collapse in agent search: the lazy likelihood-displacement. External Links: 2512.04220, Link Cited by: §6.3.
• G. Dong, H. Mao, K. Ma, L. Bao, Y. Chen, Z. Wang, Z. Chen, J. Du, H. Wang, F. Zhang, G. Zhou, Y. Zhu, J. Wen, and Z. Dou (2025) Agentic reinforced policy optimization. External Links: 2507.19849, Link Cited by: §6.3.
• K. Fan, K. Feng, M. Zhang, T. Peng, Z. Li, Y. Jiang, S. Chen, P. Pei, X. Cai, and X. Yue (2026) Exploring reasoning reward model for agents. arXiv preprint arXiv:2601.22154. Cited by: §1.
• K. Feng, K. Gong, B. Li, Z. Guo, Y. Wang, T. Peng, J. Wu, X. Zhang, B. Wang, and X. Yue (2025a) Video-r1: reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776. Cited by: §6.3.
• K. Feng, M. Zhang, S. Chen, Y. Lin, K. Fan, Y. Jiang, H. Li, D. Zheng, C. Wang, and X. Yue (2026) Gen-searcher: reinforcing agentic search for image generation. arXiv preprint arXiv:2603.28767. Cited by: §1, §6.1.
• K. Feng, M. Zhang, H. Li, K. Fan, S. Chen, Y. Jiang, D. Zheng, P. Sun, Y. Zhang, H. Sun, et al. (2025b) Onethinker: all-in-one reasoning model for image and video. arXiv preprint arXiv:2512.03043. Cited by: §6.3.
• M. Fu, Y. Peng, B. Liu, Y. Wan, and D. Chen (2025) LiveVQA: live visual knowledge seeking. arXiv preprint arXiv:2504.05288. Cited by: §3.2, §5.
• A. Garg, C. Zhang, N. Neema, D. Bick, G. Venkatesh, and J. Hestness (2026) CoRPO: adding a correctness bias to grpo improves generalization. External Links: 2511.04439, Link Cited by: §6.3.
• X. Geng, P. Xia, Z. Zhang, X. Wang, Q. Wang, R. Ding, C. Wang, J. Wu, Y. Zhao, K. Li, et al. (2025) Webwatcher: breaking new frontier of vision-language deep research agent. arXiv preprint arXiv:2508.05748. Cited by: §1, §3.1, §3.1, Table 2, Table 2, §5, §6.2.
• J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, and J. Guo (2025) A survey on llm-as-a-judge. External Links: 2411.15594, Link Cited by: §3.3, 2nd item.
• D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), pp. 633–638. Cited by: §1, Figure 3, §5.2, §6.3.
• J. Hong, C. Zhao, C. Zhu, W. Lu, G. Xu, and X. Yu (2025) DeepEyesV2: toward agentic multimodal model. arXiv preprint arXiv:2511.05271. Cited by: Table 2, §6.2.
• W. Hu, X. Chen, Y. Gao-Tian, Y. Deng, N. Peng, and K. Chang (2026) OpenVLThinkerV2: a generalist multimodal reasoning model for multi-domain visual tasks. External Links: 2604.08539, Link Cited by: §6.3.
• W. Huang, Y. Zeng, Q. Wang, Z. Fang, S. Cao, Z. Chu, Q. Yin, S. Chen, Z. Yin, L. Chen, et al. (2026) Vision-deepresearch: incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601.22060. Cited by: §B.2, Appendix C, Figure 10, Appendix E, §1, §3.1, §3.1, §3.1, §4.2, §4.2, Figure 3, §5.2, §5, §6.1, §6.3.
• [24] D. Jiang, R. Zhang, Z. Guo, Y. Wu, P. Qiu, P. Lu, Z. Chen, G. Song, P. Gao, Y. Liu, et al. Mmsearch: unveiling the potential of large models as multi-modal search engines. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §5.
• B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, Link Cited by: §A.2, Appendix A, §2, §2, §4.1, §4.2, §4.2, Figure 3, §5.2, Table 3, §6.1.
• K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. (2025a) WebSailor: navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592. Cited by: §3.1, §3.1.
• K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, W. Shen, J. Zhang, D. Zhang, X. Wu, Y. Jiang, M. Yan, P. Xie, F. Huang, and J. Zhou (2025b) WebSailor: navigating super-human reasoning for web agent. External Links: 2507.02592, Link Cited by: §6.1.
• H. Liu and L. Xiao (2025) RE-grpo: leveraging hard negative cases through large language model guided self training. Neurocomputing, pp. 132543. Cited by: §6.3.
• Z. Liu, Y. Zang, Y. Zou, Z. Liang, X. Dong, Y. Cao, H. Duan, D. Lin, and J. Wang (2025) Visual agentic reinforcement fine-tuning. arXiv preprint arXiv:2505.14246. Cited by: Table 2.
• K. Narayan, Y. Xu, T. Cao, K. Nerella, V. M. Patel, N. Shiee, P. Grasch, C. Jia, Y. Yang, and Z. Gan (2025a) Deepmmsearch-r1: empowering multimodal llms in multimodal web search. arXiv preprint arXiv:2510.12801. Cited by: §6.1.
• K. Narayan, Y. Xu, T. Cao, K. Nerella, V. M. Patel, N. Shiee, P. Grasch, C. Jia, Y. Yang, and Z. Gan (2025b) Deepmmsearch-r1: empowering multimodal llms in multimodal web search. arXiv preprint arXiv:2510.12801. Cited by: Table 2.
• OpenAI (2025) Introducing gpt-5. Note: https://openai.com/index/introducing-gpt-5/ Cited by: 3rd item, Table 2, Table 2.
• A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §D.2.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §A.1, §A.1.
• B. Seed (2026) Seed2.0 model card. External Links: Link Cited by: §1.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §A.1, Appendix A, §4.2, §6.3.
• A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. (2025) Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: §1.
• M. Song, H. Sun, J. Gu, L. Li, L. Xu, R. Krishna, and Y. Cheng (2026) AdaReasoner: dynamic tool orchestration for iterative visual reasoning. External Links: 2601.18631, Link Cited by: §6.2.
• S. Tan, M. Luo, C. Cai, T. Venkat, K. Montgomery, A. Hao, T. Wu, A. Balyan, M. Roongta, C. Wang, L. E. Li, R. A. Popa, and I. Stoica (2025) RLLM: a framework for post-training language agents. Note: Notion Blog Cited by: Appendix C, §5.
• A. Team (2025a) Claude 3.7 sonnet system card. Note: https://www-cdn.anthropic.com/9ff93dfa8f445c932415d335c88852ef47f1201e.pdf Cited by: Table 2, Table 2.
• A. Team (2025b) System card: claude opus 4 & claude sonnet 4. Note: https://www-cdn.anthropic.com/6d8a8055020700718b0c49369f60816ba2a7c285.pdf Cited by: §1, Table 2.
• A. Team (2026a) Claude opus 4.6 system card. Note: https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf Cited by: §3.3.
• K. Team (2026b) Kimi k2.5: visual agentic intelligence. External Links: 2602.02276, Link Cited by: §1.
• O. Team (2024) GPT-4o system card. External Links: 2410.21276, Link Cited by: §D.2, §D.2, §3.1, §3.3, 2nd item, Table 2, Table 2.
• S. V. Vuddanti, A. Shah, S. K. Chittiprolu, T. Song, S. Dev, K. Zhu, and M. Chaudhary (2025) PALADIN: self-correcting language model agents to cure tool-failure cases. External Links: 2509.25238, Link Cited by: §6.3.
• P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Hengel (2017) Fvqa: fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence 40 (10), pp. 2413–2427. Cited by: §1, §3.2, §5.
• Y. Wei, L. Zhao, K. Lin, E. Yu, Y. Peng, R. Dong, J. Sun, H. Wei, Z. Ge, X. Zhang, et al. (2025) Perception in reflection. arXiv preprint arXiv:2504.07165. Cited by: §6.2.
• [48] Wikipedia. Note: https://www.wikipedia.org/ Cited by: §3.1.
• J. Wu, B. Li, R. Fang, W. Yin, L. Zhang, Z. Tao, D. Zhang, Z. Xi, G. Fu, Y. Jiang, et al. (2025a) Webdancer: towards autonomous information seeking agency. arXiv preprint arXiv:2505.22648. Cited by: §3.1.
• J. Wu, Z. Deng, W. Li, Y. Liu, B. You, B. Li, Z. Ma, and Z. Liu (2025b) MMSearch-r1: incentivizing lmms to search. arXiv preprint arXiv:2506.20670. Cited by: Table 2, §6.1.
• H. Yao, Q. Yin, M. Yang, Z. Zhao, Y. Wang, H. Luo, J. Zhang, and J. Huang (2026) Mm-deepresearch: a simple and effective multimodal agentic search baseline. arXiv preprint arXiv:2603.01050. Cited by: §6.1.
• S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §2, §3.3.
• S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, et al. (2024) Visrag: vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594. Cited by: §6.2.
• Y. Zeng, W. Huang, Z. Fang, S. Chen, Y. Shen, Y. Cai, X. Wang, Z. Yin, L. Chen, Z. Chen, S. Huang, Y. Zhao, Y. Hu, P. Torr, W. Ouyang, and S. Cao (2026) Vision-deepresearch benchmark: rethinking visual and textual search for multimodal large language models. preprint. Cited by: §1, §5.
• J. Zhang, S. Zeng, K. Guo, X. Dai, H. Liu, J. Tang, and Y. Chang (2026) Fix before search: benchmarking agentic query visual pre-processing in multimodal retrieval-augmented generation. arXiv preprint arXiv:2602.13179. Cited by: §6.2.
• Y. Zhang, L. Hu, H. Sun, P. Wang, Y. Wei, S. Yin, J. Pei, W. Shen, P. Xia, Y. Peng, et al. (2025) Skywork-r1v4: toward agentic multimodal intelligence through interleaved thinking with images and deepresearch. arXiv preprint arXiv:2512.02395. Cited by: §3.1, §3.1, §6.1.
• Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma (2024) LlamaFactory: unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. External Links: Link Cited by: Appendix C, §5.
• Z. Zheng, M. Yang, J. Hong, C. Zhao, G. Xu, L. Yang, C. Shen, and X. Yu (2026) DeepEyes: incentivizing "thinking with images" via reinforcement learning. External Links: 2505.14362, Link Cited by: §6.3.
• K. Zhu, Z. Liu, B. Li, M. Tian, Y. Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, X. Ma, X. Yu, G. Ramesh, J. Wu, Z. Liu, P. Lu, J. Zou, and J. You (2025) Where llm agents fail and how they can learn from failures. External Links: 2509.25370, Link Cited by: §6.3.

Appendix

Appendix Contents

Appendix A Preliminary of Reinforcement Learning

This section details the reinforcement learning (RL) preliminaries underpinning our multi-turn training objective. We first review the standard Group Relative Policy Optimization (GRPO) algorithm (Shao et al., 2024), which operates on single-turn generations. Subsequently, we describe its extension to search-augmented rollouts, as introduced by Search-R1 (Jin et al., 2025), wherein the policy interleaves generation with calls to an external search engine. These formulations serve as the direct precursors to the multi-turn, multi-tool objective employed by OpenSearch-VL (see Sec. 4.2).

A.1 Standard GRPO

GRPO (Shao et al., 2024) is an actor-critic variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017) that eliminates the reliance on a learned value function. Instead of maintaining a separate critic model to estimate a baseline, GRPO derives its advantage estimate by comparing multiple responses sampled for the same prompt, collectively defining a group. This parameter-efficient design is particularly advantageous in settings where only a scalar outcome-level reward is available, reducing memory overhead and mitigating the instability of training a dense token-level value function.Formally, given a prompt distribution and a reference policy , GRPO samples a group of candidate responses for each prompt . Each response is evaluated via a reward function . To compute the advantages, GRPO standardizes these rewards within the local group:

(13)

By this formulation, all constituent tokens of a given response are assigned a uniform advantage scalar. The policy parameters are subsequently optimized by maximizing the clipped surrogate objective:

				(14)

where the importance sampling ratio is defined as

(15)

Here, represents the probability ratio clipping hyperparameter, and modulates the Kullback–Leibler (KL) divergence penalty against a fixed reference policy . In contrast to standard PPO—which subsumes the KL penalty directly into the reward signal and necessitates a parameterized value network to approximate (Schulman et al., 2017)—GRPO enforces the KL regularization explicitly within the loss landscape. By deriving strictly from empirical group statistics (Eq. 13), GRPO naturally aligns with the comparative structure of preference-based reward modeling.

A.2 GRPO with Search Engine

Search-R1 (Jin et al., 2025) extends the GRPO framework from unimodal, single-turn generation to an interleaved, search-augmented generative process. Rather than sampling an isolated response , Search-R1 generates a multi-step rollout comprising both policy-emitted tokens and external evidence retrieved from a search engine . This search-augmented sampling distribution is denoted as:

(16)

where represents the interleaving operator: whenever the policy emits a search command, the retrieved documents are appended to the conditioning context, and the policy resumes generation conditioned on this expanded prefix.To prevent the optimizer from erroneously updating parameters based on exogenous environmental tokens, Search-R1 utilizes a masking function that filters out non-generated tokens. Defining as an indicator variable where if is explicitly generated by the policy, the corresponding Search-R1 GRPO objective becomes:

				(17)

featuring the environment-conditioned importance ratio:

(18)

Relative to standard GRPO (Eq. 14), this adaptation introduces two pivotal modifications. First, the expectation is calculated with respect to the interleaved distribution , ensuring the causal conditioning history incorporates all previously retrieved external evidence. Second, the objective is normalized strictly by the count of generated tokens, , restricting gradient calculations to policy-authored positions. Equivalently masking the KL divergence ensures the model is not penalized for distributional divergence over environmental observations. These mechanisms adapt GRPO to tool-in-the-loop architectures and establish the theoretical foundation for our multimodal environment formulation (Sec. 4.2).

Appendix B Multi-Turn Search Fatal-Aware GRPO Details

Building upon the foundations established in Appendix A, this section details the mechanics of our multi-turn, search-augmented RL objective introduced in Sec. 4.2. We articulate the detection logic for fatal execution cascades and rigorously derive the advantage estimation process incorporating one-sided clamping; the composite reward and its three components are defined directly in Sec. 4.2.

B.1 Fatal Step Detection Logic

Unconstrained interactions with external environments inevitably yield “fatal” states—irrecoverable error cascades rendering subsequent reasoning invalid. We articulate the procedure for computing the fatal step index , which drives the fatal-aware token mask (Eq. 10).For a given trajectory of length , we instantiate a stateful error counter initialized to zero. At each sequential step , the counter transitions according to:

(19)

The fatal step index is subsequently defined as the earliest step where the error threshold is breached:

(20)

In our implementation, we set . If the threshold is never reached, the trajectory is classified as non-fatal, denoted by setting .This formulation embodies two critical design properties. First, the explicit reset condition () ensures that isolated transient errors—which are ubiquitous and often recoverable in realistic web environments—do not prematurely abort the trajectory. Second, the conservative threshold () dictates that a trajectory is only deemed fatal following three consecutive failures, affording the autoregressive policy a robust opportunity to self-correct prior to gradient masking. Tool-execution errors triggering this counter encompass timeouts, malformed API payloads, and argument-parsing failures; these are flagged deterministically by the execution sandbox, independent of any learned heuristic.

B.2 Advantage Derivation with One-Sided Clamping

We rigorously detail the advantage estimation protocol underpinning the objective function in Eq. 12.

Step 1: Unbiased Group Statistics.

For a sampled group of rollouts originating from the identical prompt , the empirical group mean and standard deviation are computed over the composite rewards:

(21)

Here, denotes the stop-gradient operator. Consequently, and act as static normalizing constants during backpropagation. Gradients flow exclusively through the importance sampling ratios evaluated on the unmasked, policy-generated tokens.

Step 2: Universal Group Normalization.

Each trajectory within the group is assigned a standardized reward:

(22)

where provides numerical stability. Crucially, all trajectories—expressly including those truncated by the fatal condition ()—are incorporated into the calculation of and . This deliberate inclusion ensures that fatal trajectories actively shape the group-level baseline, anchoring the relative performance ranking across the entire sampled cohort.

Step 3: Fatal-Aware One-Sided Clamping.

The standardized rewards are mapped to the final advantage estimates via a piecewise clamping function:

(23)

Step 4: Gradient Dominance over Hard-Masking.

We now formalize the informal claim that fatal-aware clamping strictly dominates the hard-masking baseline of (Huang et al., 2026) in gradient informativeness. Let

(24)

denote the (un-clipped) per-trajectory contribution to under our scheme, and let denote the corresponding contribution under the hard-masking baseline, which discards every fatal trajectory in full so that whenever . We compare the two on the support of fatal trajectories.

Proposition 1(Dominance over hard-masking).

Fix any fatal trajectory (). Then:

1. If , then .

2. If , then , while coincides with the search-augmented GRPO gradient (Eq. 17) of restricted to its viable prefix , evaluated at the non-negative advantage .

Consequently, is weakly informative-dominant over : it never propagates gradient through the post-fatal suffix, never penalises the viable prefix, and strictly extracts positive reinforcement on prefixes whose group-normalised return exceeds the baseline.

Proof.

By Eq. 11, implies , so the surrogate term and its gradient vanish identically in Eq. 24; combined with by definition of hard-masking, this establishes (i). For , , while the fatal-aware mask retains exactly the policy-generated tokens of the viable prefix and zeros out (a) environment-emitted tokens and (b) all post-fatal tokens. On this support the per-token integrand of Eq. 24 reduces token-by-token to that of Eq. 17 with the same importance ratio and advantage , so the per-token surrogates—and hence their gradients—agree. Since by definition, this establishes (ii).∎

In aggregate, the one-sided clamp safely ignores invalid credit assignment while selectively harvesting positive reinforcement from prematurely truncated yet high-quality exploratory rollouts.

Bias of Group Statistics under Clamping.

The asymmetric definition of deserves an honest accounting. We compute in Step 1 over all rollouts—including fatal ones—so the standardised score is zero-mean within the group; Step 3 then maps any negative on a fatal trajectory to zero. Letting index the fatal subset of the group,

(25)

so the clamp induces a non-negative bias relative to the zero-mean GRPO baseline, in exchange for non-zero gradient on viable prefixes. We argue this trade-off is benign in our regime for two reasons. (a) Concentration of fatal rollouts in the lower mode. Figure 5 shows that of fatal rollouts have and are clamped to zero, while the residual that survive sit in the same score regime as competitive non-fatal rollouts; is therefore dominated by samples whose viable prefix is empirically high-quality, rather than by indiscriminate inflation of low-quality fatals. (b) Preservation of relative ranking. Because shifts only the fatal subset upward and never displaces non-fatal advantages, the within-group ordering between non-fatal trajectories—which is the actual signal GRPO exploits—remains intact. We therefore interpret as a controlled positive bias that buys back gradient on prematurely truncated yet promising prefixes; replacing the clamp with an unbiased estimator (e.g., a separate baseline computed only over non-fatal rollouts, or a doubly-robust correction for ) is an interesting alternative we leave for future work.A sanity check of the resulting clamping behaviour, including a per-group-difficulty breakdown and the aggregate score distribution over groups, is reported in Sec. 5 (Figs. 4 and 5).

Appendix C Implementation Details

Our agentic SFT pipeline builds on LlamaFactory (Zheng et al., 2024), which we extend with multi-turn, tool-interleaved data collators and a Qwen3-VL-aware vision/text packing scheme so that interleaved image observations produced by visual tools (Crop, Sharpen, SuperResolution, PerspectiveCorrect, OCR) can be consumed verbatim by the policy. Our RL pipeline builds jointly on rLLM (Tan et al., 2025) and Vision-DeepResearch (Huang et al., 2026): from the former we adopt the asynchronous agent rollout architecture (decoupled trajectory generation, replay, and policy-update workers), and from the latter we adopt the multimodal trajectory abstraction. On top of these two, we implement a Qwen3-VL chat-template renderer, an interleaved image-token re-alignment routine for variable-length visual observations, and an asynchronous multi-turn rollout engine that supports tool-call interruption and resumption. The resulting pipeline is what we use to train all OpenSearch-VL variants in this paper.

C.1 SFT Training Configuration

媒体内容 · 前往原文查看

Category	Hyperparameter	Value / Setting
Model	Base Model	Qwen3-VL-8B-Instruct (32B, 30B-A3B)
	Image Max Pixels	()
	Video Max Pixels	()
	Trust Remote Code	True
Method	Finetuning Type	Full
	Vision Tower Frozen	False
	MM Projector Frozen	False
	DeepSpeed Stage	ZeRO-3
	Mixed Precision	bfloat16
Dataset	Total Samples	36592
	Template	qwen3_vl
	Cutoff Length	tokens
	Preprocessing Workers	16
	Dataloader Workers	4
Training	Batch Size per Device	1
	Gradient Accumulation Steps	1
	Effective Batch Size	256 ()
	Gradient Checkpointing	True
	Learning Rate
	Epochs	8
	LR Scheduler	cosine
	Warmup Ratio	0.1
Infrastructure	Total GPUs	256 (32 nodes 8)
	Orchestration	Ray + DeepSpeed ZeRO-3
	Placement Strategy	PACK
	Resources per Worker	1 GPU
Logging / I O	Logging Steps	5
	Checkpoint Save Steps	400
	Plot Loss	True
	Report Backend	TensorBoard

Table 4 reports the full hyperparameter configuration used for the agentic supervised fine-tuning stage across all three OpenSearch-VL variants. The three model sizes (8B dense, 32B dense, 30B-A3B Mixture-of-Experts) share identical training hyperparameters—only the base checkpoint path differs—so we present them in a single consolidated table. All runs use full-parameter finetuning (including the vision tower and multi-modal projector) with DeepSpeed ZeRO-3 and Ray-based orchestration over 256 GPUs (32 nodes 8 GPUs).

C.2 RL Training Configuration

Table 5 reports the key hyperparameters for the multi-turn fatal-aware GRPO stage across the two OpenSearch-VL variants we RL-finetune: 8B dense (Qwen3-VL-8B-Instruct) and 30B-A3B MoE (Qwen3-VL-30B-A3B-Instruct). Both runs use the same async SGLang rollout engine and the same Megatron-parallel actor, with a shared algorithm template (RLOO leave-one-out advantage under the GRPO group-relative objective, low-variance KL controller, no critic). We list only the fields set explicitly in our launch scripts; framework defaults (e.g. optimizer betas, reward-model paths, checkpoint-resume flags) are omitted. Rows whose value is identical for both variants are written once; rows that differ are split into two cells.

媒体内容 · 前往原文查看

Category	Hyperparameter	8B (dense)	30B-A3B (MoE)
Model	Base Model	Qwen3-VL-8B-Instruct	Qwen3-VL-30B-A3B-Instruct
	Training Dtype	bfloat16
Data	Train Batch Size (prompts)	256
	Val Batch Size	64	512
	Max Prompt Length	tokens
	Max Response Length	tokens
	Data Seed	3407
Rollout	Engine	SGLang (async mode)
	Rollout Tensor Parallel	4
	# Samples per Prompt ()	8	16
	GPU Mem. Utilization	0.85	0.65
	Train / Val Temperature	0.7 / 0.7
	Train / Val Top-	1.0 / 0.95
	Top-	(disabled)
Policy (Actor)	Strategy	Megatron-LM
	Tensor Parallel (TP)	4
	Pipeline Parallel (PP)	2
	Context Parallel (CP)	8	4
	Expert Parallel / ETP	—	8 / 1
	PPO Mini-batch Size	64	128
	PPO Max Token Len / GPU
	Micro-batch Size / GPU	1
	Dynamic Batch Size	True
	Param / Optim / Grad Offload	CPU
	Gradient Checkpointing	Full recompute, uniform (1 layer)
Optim. / Loss	Actor LR
	PPO Clip Ratio (high)	0.28
	Entropy Coefficient	0.0
	Use KL Loss	False
	KL Loss / Controller Coef	/
	Loss Aggregation	seq-mean-token-sum
Algorithm	Advantage Estimator	RLOO (within GRPO objective)
	KL Type	low-variance KL
	Fatal-aware Masking	True (unknown + error)
Tool-Agent	# Parallel Tasks	256
	# Parallel Tool Calls
	Stepwise Advantage	False
Trainer	Cluster	8 nodes 8 GPUs = 64 GPUs
	Save / Test Freq (steps)	50 / 10	1 / 10
	Total Epochs	100	5
	Critic Warmup	0 (critic-free)
MoE-only (30B-A3B)	Router Dtype	—	fp32
	Auxiliary-Loss Coef	—	0.01
	Router Z-loss Coef	—
	Permute Fusion	—	True

Appendix D Data Curation Details

This section complements Sec. 3.1 with the full operational specification of the multi-hop VQA construction pipeline and traces the pipeline end-to-end through a concrete seed page. All statistics below are computed on the 2025-05-01 snapshot of English Wikipedia.

D.1 Path Sampling Hyperparameters

We fix on the in-degree measured against the snapshot. Any candidate node whose incoming count exceeds is skipped, rejecting roughly the top of all article nodes. This cut-off excludes continent-, country-, and century-level pages (e.g. United_States, Queensland, 21st_century, English_language) for which the uniqueness invariant in Eq. 6 is routinely violated, while preserving the long tail of entity pages that carry substantive semantic content.

Path length distribution.

Path lengths are sampled as . Shorter paths are favoured in order to keep the downstream rollout horizons tractable; the upper bound is set empirically, as walks longer than four hops rarely survive the uniqueness and non-leakage checks of Eq. 6 without heavy resampling.Beyond the three rules stated in the main text, the walk additionally skips:

1. any title containing the substring (disambiguation), or beginning with List of, Outline of, Index of, or Timeline of;

2. all non-article namespaces, i.e. Template:, Category:, File:, User:, Help:, Portal:, Wikipedia:;

3. redirect pages: each outgoing link is first dereferenced to its target article, and the exclusion rules above are applied to the dereferenced target rather than the surface link.

Seeds are drawn by stratified sampling across five coarse domains—Person, Building/Place, Location (non-hub), Organism, and Artifact—to balance the representation of visually groundable categories. A node is eligible as a seed iff it (i) exposes an infobox; (ii) links to at least one Wikimedia Commons image of resolution no smaller than ; and (iii) has in-degree in , so that the seed is neither a dead end nor a hub.A walk rooted at a fixed seed is retried up to times whenever it (i) hits a hub or a filtered namespace, (ii) fails to produce any descriptor for some bridge that survives the LLM uniqueness evaluator, or (iii) terminates at a node whose infobox contains no attribute meeting the six-token length bound. Seeds that exhaust all attempts are dropped from the final pool.

D.2 Running Example: Australia_Zoo

To make the five stages of Sec. 3.1 concrete, we trace the pipeline end-to-end on the seed page Australia_Zoo with .The outgoing links of Australia_Zoo are first partitioned by the exclusion rules of Appendix D.1. Representative entries of each partition are:

• Retained (entity pages, in-degree below ): Bob_Irwin, Steve_Irwin, Terri_Irwin, Bindi_Irwin, Robert_Irwin, The_Crocodile_Hunter, Wildlife_Warriors, Beerwah,_Queensland, Rosedale,_Queensland, Angkor_Wat, …

• Hub-rejected (in-degree ): Queensland, Brisbane, Australia, United_States, Zoo.

A random walk with then samples

From the lead paragraph of , we extract the attribute date of Australian citizenship, yielding the short answer (3 tokens). GPT-4o (Team, 2024) is then prompted with the full path and its relations to synthesize the canonical question

		“On what date did Terri Irwin, the wife of Steve Irwin—the man who
		took over management of Australia Zoo in 1991—become an Australian citizen?”

Rewriting proceeds from the farthest bridge toward . Table 6 lists the descriptor candidates proposed by GPT-4o (Team, 2024) from ’s Wikipedia context and the verdicts returned by the uniqueness evaluator. The first accepted descriptor is retained (ties broken uniformly at random).

媒体内容 · 前往原文查看

Descriptor for	Uniqueness verdict
“the man who took over management of [] in 1991”	✓ unique
“the wildlife documentary host killed by a stingray in 2006”	✓ unique
“the son of the founders of []”	– Bob and Lyn had three children (Joy, Steve, Mandy)

The accepted descriptor produces the fuzzy form

		“On what date did the wife of the man who took over management of
		Australia Zoo in 1991 become an Australian citizen?”

All three invariants of Eq. 6 are satisfied: is preserved, the descriptor combined with “wife of” resolves uniquely to Terri Irwin, and contains no surface form or alias of any node along (e.g. neither Steve Irwin, Terri Irwin, nor The Crocodile Hunter appears in ).We then retrieve candidate images from Wikimedia Commons under the query Australia_Zoo and rank them by CLIP (Radford et al., 2021) cosine similarity to the canonical description “Australia Zoo entrance”. The top-ranked image (Fig. 6, similarity ) is selected as . Replacing the anchor mention in with a visual referring expression yields the final VQA instance

The instance passes all four checks of Sec. 3.1: masking ( contains no entity name or alias from ); uniqueness (a GPT-4o judge given only returns exactly one consistent answer); visual relevance (CLIP similarity exceeds our threshold); and non-triviality, verified jointly under the filtering process of Sec. 3.2.

D.3 Counterfactual Design Choices

We probe the necessity of the three key design choices in Sec. 3.1 by counterfactually relaxing each one on the running example above.

Anchor Answer.

Replacing with a photograph of Terri Irwin and asking for her citizenship date reduces the task to a single reverse-image lookup: ImageSearch on the new returns the Wikipedia page of Terri Irwin directly, from which the citizenship date is read off in one step. The multi-hop structure collapses, independently of how the question is phrased.

No fuzzing.

Retaining the canonical as the final question—i.e. skipping the fuzzy-rewriting stage—exposes all entity names in plain text, and a single TextSearch(“Terri Irwin Australian citizenship date”) suffices to recover . The image becomes decorative rather than load-bearing.

No hub avoidance.

If the walk were allowed to admit the hub Queensland as a bridge, a natural descriptor such as “the zoo located in []” would fail the uniqueness check—thousands of zoos satisfy this relation—forcing either unsuccessful resampling or a contrived descriptor that itself leaks the identity of Australia_Zoo. Hub avoidance (Appendix D.1) removes this failure mode at sampling time rather than relying on the downstream filters to catch it.

Appendix E System Prompts

This section provides the complete system prompts used in OpenSearch-VL. The agent system prompt (Figure 7) is shared across inference and SFT data collection, and is paired with the machine-readable tool schema (Figure 8) that is injected into the model context as <tools>...</tools> for OpenAI-style function calling. The two reward judge prompts (Figures 8 and 11) are used during RL training to compute and , respectively. For final benchmark reporting, we additionally adopt a GPT-4o judge prompt (Figure 10) that is aligned with the evaluation protocol of Vision-DeepResearch (Huang et al., 2026), so that our numbers remain directly comparable to prior multimodal deep-research work.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Appendix F Tool Definition and Usage

This section details the search-oriented tools integrated within OpenSearch-VL. For each tool, we articulate its core functionality, formalize its input–output signature, describe its backend implementation, and contextualize its operational role within visual search trajectories. The tool set is partitioned into three functional modalities: Retrieval, Image Enhancement, and Attention & Parsing. Retrieval tools interface with the open web via remote APIs; image enhancement tools operate as lightweight, deterministic local primitives; and attention & parsing tools combine local spatial priors with remote document-understanding services.

Retrieval Tools

• TextSearch

  • –

    Functionality: A composite textual retrieval mechanism comprising three pipelined stages: (i) a Serper-backend search query to retrieve top- candidate URLs; (ii) a JINA Reader invocation to fetch and normalize the HTML payloads into clean Markdown; and (iii) a Qwen3-32B summarization pass that distills a query-focused 2–4 sentence synopsis per document. The terminal output is a structured list of [Passage ] (title, url, summary) objects.

  • –

    Operational Role: Functions as the primary “read the web” primitive when surface-level snippets are insufficient. It is indispensable for resolving complex, multi-hop knowledge queries and retrieving long-tail factual evidence that necessitates reasoning over entire paragraphs.

• ImageSearch

  • –

    Functionality: A visual-entity and reverse-image search tool powered by the Polaris Lens API. It accepts a publicly routable image URL and returns a structured JSON payload encompassing visually similar images, recognized entities, related domain URLs, and textual captions associated with the query image.

  • –

    Operational Role: Serves as the critical bridge transmuting purely visual queries into retrievable textual entities. When the agent determines that resolving a query hinges on identifying an unknown visual entity (e.g., a landmark, logo, or public figure), it invokes ImageSearch to extract external semantic grounding, which can subsequently be cross-referenced via TextSearch.

Image Enhancement Tools

• Sharpen

  • –

    Functionality: A deterministic deblurring operator implemented via OpenCV Unsharp Masking. Parameterized by an input image and an optional sharpening intensity (default ), it computes the enhanced image , where is a Gaussian blur kernel.

  • –

    Operational Role: Deployed when the input image exhibits pervasive blur or soft edge gradients, which frequently degrade downstream optical character recognition (OCR) or object detection. As a computationally cheap, side-effect-free preprocessing step, the agent can heuristically apply Sharpen prior to reinvoking perceptual tools.

• SuperResolution

  • –

    Functionality: A deep-learning-based upscaling tool employing the EDSR architecture via OpenCV’s dnn_superres module. It accepts an image and a discrete scale factor (default ), emitting a high-resolution reconstruction. For robustness, if the EDSR weights are unavailable in the deployment environment, it gracefully degrades by returning the original image.

  • –

    Operational Role: Crucial for mitigating resolution bottlenecks, particularly on tightly cropped patches or inherently low-fidelity inputs (e.g., thumbnails). By synthesizing high-frequency details, it substantially elevates the reliability and extraction accuracy of subsequent OCR invocations.

• PerspectiveCorrect

  • –

    Functionality: An automated perspective-rectification primitive. It executes a Canny edge detection pipeline on the grayscale projection, extracts the maximal quadrilateral contour, and computes a four-point perspective transform to warp the image into a fronto-parallel plane. If a reliable quadrilateral cannot be established, it falls back to the original image alongside a diagnostic warning.

  • –

    Operational Role: Addresses the pervasive domain shift of real-world captures, such as skewed photographs of documents, receipts, or screens. Rectifying the image geometry dramatically improves the bounding-box precision and text-recognition fidelity of downstream OCR parsing.

Attention and Parsing Tools

• Crop

  • –

    Functionality: A deterministic spatial-attention primitive. Parameterized by an image and a bounding box coordinate tuple , it extracts and isolates the specified rectangular sub-region, writing the artifact to disk for subsequent reference.

  • –

    Operational Role: Models the human cognitive mechanism of foveating onto a dense sub-region within a cluttered scene (e.g., isolating a single chart in a multi-panel figure). It is the canonical mechanism for the policy to suppress peripheral noise prior to passing a clean, localized patch to downstream tools such as OCR, SuperResolution, or ImageSearch.

• OCR

  • –

    Functionality: A layout-aware optical character recognition service backed by a remote PaddleX infrastructure. It consumes a base64-encoded image alongside optional flags for chart-recognition and document-orientation classification. It emits a hierarchical list of detected text blocks (categorized into titles, body text, footnotes, etc.), yielding explicit block_label, block_content, and a reading-order-reconstructed formatted_text.

  • –

    Operational Role: Serves as the agent’s primary “read the image” capability. Beyond plain character recognition, its preservation of the document’s logical layout is indispensable for queries dependent on structural hierarchy (e.g., distinguishing a figure caption from a section title). It is optimally invoked following upstream enhancements (PerspectiveCorrect, Sharpen, SuperResolution) or spatial isolation (Crop).

Appendix G Case Study

Figure 9 presents a representative end-to-end trajectory of OpenSearch-VL on a knowledge-intensive visual question (“In what year did this bridge open?”), whose answer is recoverable neither from the encoder’s parametric knowledge nor from any single retrieval call. The agent first dispatches a Crop on the roadside signage to foveate on the most diagnostic sub-region, forwards the cropped patch to ImageSearch to identify the structure as the Kessock Bridge, and then issues a targeted TextSearch that corroborates the opening year as 1982. The trace exemplifies the compositional “verify, don’t guess” behavior that the visual–retrieval action space , the query-quality reward , and the fatal-aware masking scheme are jointly designed to incentivize: tools are chained in an order that progressively grounds the question, and the rollout terminates as soon as cross-modal evidence converges.