fazii@mail.ustc.edu.cn, wxhuang0616@gmail.com

Abstract

We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution. To address these challenges, we propose Video-DeepResearch, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe—supervised fine-tuning followed by Group Relative Policy Optimization (GRPO)—enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate VideoDR-Bench, a human-AI collaborative benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale.

Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

Video-Deepresearch Team fazii@mail.ustc.edu.cn, wxhuang0616@gmail.com https://github.com/Osilly/Vision-DeepResearch

1 Introduction

Executing long-horizon tasks through active interaction marks a critical milestone in the pursuit of AGI, a capability epitomized by the recent rise of deep research agents Li et al. (2025b); Wu et al. (2025a); Tao et al. (2025). Leveraging advanced VLMs Bai et al. (2025); Team (2026); Comanici et al. (2025), research agents have advanced into unstructured, vision-rich digital wildernesses Chen et al. (2026); Huang et al. (2026); Ma et al. (2025); Wu et al. (2025b); Feng et al. (2026). This transition shifts the agent’s workload from processing clean textual inputs to decoupling critical insights from noisy, heavily redundant multimedia web layouts, redefining the complexity of autonomous exploration.

Previous literature addressing these challenges generally tracks a multi-modal trajectory. Text-based web agents, notably WebGPT Nakano et al. (2021) and AutoGPT Yang et al. (2023), established the foundations of iterative knowledge synthesis through programmatic search. To accommodate rich visual layouts, subsequent paradigms shifted toward sensory interfaces; frameworks like WebWatcher Geng et al. (2025) introduced various vision tools for better search, and recent Vision-DeepResearch Huang et al. (2026) expanded this frontier to execute multi-step, exhaustive search across highly dense and complex visual contexts.

Despite these advances, current literature bypasses an ecologically valid yet far more formidable setting: Video-DeepResearch (Video-DR). This paradigm shifts the focus from isolated modalities to a holistic environment where text-based synthesis and dense visual tracking are deeply intertwined, fundamentally redefining the cognitive workload of autonomous agents. Pioneering this Video-DR frontier entails two fundamental bottlenecks that existing methodologies fail to address. First, on the data synthesis front, it remains largely elusive how to construct effective training pipelines that align with the intrinsic properties of video-based deep research. Conventional video datasets focus on localized captions or short-term action labels; conversely, Video-DR requires generative data curation that couples long-horizon decision trajectories with dense, time-varying textual and visual evidence. Second, from an evaluation perspective, establishing a rigorous and high-fidelity benchmark presents a formidable challenge. Standard visual question-answering metrics are insufficient for measuring an agent’s multi-step strategic execution, creating an urgent need for multi-dimensional evaluation protocols that can accurately quantify long-term reasoning consistency and error-recovery behavior under continuous temporal dynamics.

To tackle these challenges, we first conduct an empirical investigation into the fundamental failure modes of current agents when directly applied to Video-DR. By evaluating three representative models on an existing benchmark Liu et al. (2026), we uncover two striking findings: (i) severe modality bias, where even the strongest open-source model averages only 0.10 visual tool invocations per task while heavily relying on text search (1.27 calls), indicating a systematic aversion to active visual exploration; and (ii) parametric knowledge leakage, where GPT-5 attains a competitive accuracy of 57% with virtually zero tool calls, suggesting that existing evaluations are largely solvable through memorized world knowledge alone. These findings motivate a holistic rethinking of both the training and evaluation paradigms for Video-DR agents.

Building on these insights, we propose Video-DeepResearch, a unified framework that jointly addresses the data, training, and evaluation challenges of Video-DR. Our contributions are summarized as follows:

• A scalable data engine that produces 30K video-grounded QA pairs and 7K curated trajectories through a decoupled perception-exploration pipeline with stage-wise tool unlocking, directly countering the modality bias revealed in our preliminary study.

• A two-stage training recipe combining supervised fine-tuning with GRPO, enabling compact models to outperform far larger proprietary systems. Our Video-DeepResearch-35B-A3B (64.0%) establishes a new state-of-the-art, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 points; the 30B variant (59.3%) achieves competitive performance with the proprietary baseline while demonstrating strong open-source capability.

• VideoDR-Bench, a 200-instance multi-hop VQA benchmark built via scalable human–AI collaborative annotation, where every question provably requires both visual search and external knowledge reasoning.

2 A Naive Attempt: From Image DeepResearch to Video DeepResearch

As shown in Fig. 1, we conceptualize video as a temporal composition of key entity trajectories. Consequently, our key insight is that Video-DR can be formulated as a sequential grounding pipeline: identifying critical temporal frames, performing localized visual search, and synthesizing findings via subsequent text retrieval. To operationalize this workflow and bridge the gap from image-centric research, we equip the agent with two fundamental tools: Select_Keyframe and Crop_Search. Specifically, the agent first uses Select_Keyframe to isolate informative moments from the continuous stream, and then applies Crop_Search on salient entities to construct precise visual queries, which firmly anchors the downstream text-based exploration.

Problem Formulation

Formally, given a complex research query $Q$ and a visual input $V=\{v_1,v_2,\dots ,v_T\}$ (which can be either a continuous video stream or a sequence of pre-sampled keyframes), the goal of a Video-DR agent is to synthesize a comprehensive response $R$. We formulate this as a sequential decision-making process. At step $i$, the agent generates an action $a_i\in 𝒜$ based on the historical trajectory ${\mathscr{H}}_i=[Q,V,a_1,o_1,\dots ,a_{i-1},o_{i-1}]$:

$a_i\sim {\pi }_{\theta }\left(a\mid {\mathscr{H}}_i\right)$

(1)

where ${\pi }_{\theta }$ denotes the multi-modal policy and $o$ is the execution observation from the environment. To operationalize the spatiotemporal grounding pipeline, the action space $𝒜$ encompasses our defined tools alongside standard web operations: $𝒜=\{\text{Select\_Keyframe},\text{Crop\_Search},\text{Text\_Search},\dots \}$. Specifically, the temporal tool selects a specific index $t\in \{1,\dots ,T\}$ to isolate an informative frame $v_t$. Subsequently, the spatial tool takes $v_t$ and a bounding box $B\in {ℝ}^4$ to crop a target entity, yielding a localized visual context $c_{vis}=\text{Crop}(v_t,B)$ for downstream search. Alternatively, these two tools can be encapsulated into a single joint operation.

Empirical Study

To empirically evaluate this naive formulation, we instantiate the agent with three representative models on a VideoDR Liu et al. (2026) benchmark, specifically tracking the execution frequencies of each tool to analyze their behavioral patterns.

媒体内容 · 前往原文查看

Model	Acc.	Vision Tool	Text Tools
Qwen3.5-35B-A3B	41	0.04	0.58
Qwen3.5-397B-A17B	58	0.10	1.27
GPT-5	57	0.00	0.12

As shown in Table 1, our preliminary evaluation yields two critical findings:

Finding 1: Severe Modality Bias and Visual Tool Aversion. Current models exhibit an inherent reluctance to invoke visual tools. Even the most capable open-source model, Qwen3.5-397B-A17B, executes an average of only 0.10 visual operations per task, overwhelmingly favoring text tools (1.27). This indicates that agents predominantly default to textual search, completely bypassing the intended active visual exploration.

Finding 2: Susceptibility to Parametric Knowledge Leakage. Existing evaluation setups suffer from severe prior knowledge leakage. Notably, GPT-5 achieves a highly competitive score of 57 while making virtually zero tool calls (0.00 for vision and 0.12 for text). This suggests that the model bypasses the multi-step grounding process entirely, relying solely on its vast internal memory to hallucinate or directly guess the correct answers.

3 Video-DeepResearch

We first detail a generative pipeline for synthesizing Video QA data (Sec. 3.1), which serves as the foundation for constructing high-quality execution trajectories (Sec. 3.2). Building on this curated data, we then describe the multi-modal training procedure for our agent (Sec. 3.3), and finally establish a robust benchmark to evaluate Video-DR capabilities (Sec. 3.4).

3.1 VQA Generation

Given the absence of dedicated datasets for Video-DR, we initiate our pipeline by synthesizing foundational Video QA pairs. This procedure transforms raw videos into explicit VQA pairs. To ensure high data quality, we annotate intermediate evidence for each instance, thereby ensuring the visual groundability and answerability of the generated queries.

Step 0: Multi-Domain Video Filtering. We begin by curating a diverse collection of raw videos across multiple domains. These videos are sourced from both established video datasets Ben-Ami et al. (2025); Ataallah et al. (2025); Wu et al. (2024); Wang et al. (2025); Tao et al. (2026); Li et al. (2024, 2025a); Goel et al. (2026); Li et al. (2026); Cheng et al. (2025a); Fu et al. (2025a); Hu et al. (2025); Yang et al. (2025a); Hong et al. (2026); Yang et al. (2025b); Wang and Yang (2026) and real-world streaming platforms¹¹1YouTube. These videos subsequently undergo a two-stage filtering process. First, a rule-based filter discards instances falling outside predefined duration thresholds. Second, we introduce an agentic filtering stage where Qwen3.5-35B-A3B Team (2026) assesses content complexity, eliminating videos that are uninformative or overly simplistic. Subsequently, this curated dataset is partitioned into a training set for constructing training trajectories and a test set for establishing the evaluation benchmark.

Step 1: Keyframe Selection and Visual Search. We subsequently initiate an agent-driven metadata curation process. Specifically, after proposing candidate frames via CLIP-based inter-frame similarity, we deploy Qwen3.5-397B-A17B Team (2026) to finalize the selection of keyframes $v_t$. For each $v_t$, the same model predicts bounding boxes $B$ to localize distinct entities $e$. These entities are then cropped and used to execute visual search queries. To ensure data fidelity, a secondary model (Qwen3.5-35B-A3B) verifies the semantic alignment between the cropped region and the retrieved results. Upon successful verification, we compile the video metadata, structured as a tuple: $⟨v_t,B,\text{entity name},\text{search summary}⟩$.

Step 2: VQA Generation and Verification. Finally, we synthesize QA pairs from the curated metadata via two generation patterns: (1) Single-entity: sampling one entity to formulate fact-based questions, and (2) Multi-entity: sampling $n$ entities to construct compositional questions requiring cross-entity reasoning. In both settings, we explicitly penalize superficial visual attribute queries. Post-generation, we rigorously filter out instances prone to parametric memory leakage. Specifically, we conduct four tool-free rollouts for each question; if the agent answers correctly in any attempt, the instance is permanently discarded, guaranteeing that the remaining tasks strictly require external tool utilization. Ultimately, we obtained 30k vqa pairs.

3.2 Trajectory Generation

Given the synthesized VQA pairs, we proceed to construct execution trajectories for agent training. As observed in Sec. 2, current Video-DR agents exhibit an inherent reluctance to invoke visual tools, often bypassing them in favor of text-only search. To overcome this modality bias, we propose a decoupled trajectory construction pipeline that explicitly separates visual perception from web exploration. Specifically, we generate trajectories using Qwen3.5-397B-A17B and apply rejection sampling. To operationalize the decoupling, we employ a stage-wise tool unlocking strategy. In the initial phase, the agent is restricted to a vision-only action space, comprising solely Select_Keyframe and Crop_Search. Instead of rushing to an answer, the agent is forced to execute extensive visual retrieval by cropping distinct entities across multiple keyframes. Once the agent determines the visual context is sufficient—or a predefined maximum perception horizon is reached—we expand the action space to include textual tools (Search and Visit) and prompt the agent to derive the final answer. This two-stage paradigm compels the model to conduct exhaustive cross-frame, cross-entity visual grounding prior to web exploration. Finally, we retain only the successfully resolved trajectories for downstream policy training. Ultimately, we obtained 7k correct trajectories.

3.3 Training

We adopt a two-stage training paradigm. We select Qwen3-VL-30B-A3B-Instruct Bai et al. (2025) as our base model for Video-DeepResearch-30B-A3B, given its widespread adoption as an open-source VLM and its robust AI infrastructure support. For Video-DeepResearch-35B-A3B, we adopt Qwen3.5-35B-A3B Team (2026) as the foundation model, following the same training recipe. Both variants undergo identical training procedures. In the first stage, we perform Supervised Fine-Tuning (SFT) to establish a cold start. This phase aims to align the model with the desired decoupled perception-exploration workflow, enabling it to internalize the correct multi-modal reasoning syntax. In the second stage, we apply Group Relative Policy Optimization (GRPO) to further refine the policy. By actively generating rollouts and receiving rewards for successful trajectories, the agent is encouraged to autonomously explore the action space, thereby surpassing the performance ceiling of the initial SFT phase. Experiments are conducted on a compute cluster comprising four NVIDIA H800 (80GB) GPU nodes. More details are detailed in Appendix. B.

SFT

For the SFT phase, we utilize the 7K high-quality trajectories synthesized in Sec. 3.2, enabling the model to internalize the decoupled perception-exploration paradigm. Furthermore, to address the under-utilization of text tools observed in Table LABEL:tab:_motivation, we augment our training corpus with an additional 7K text-only QA instances from VDR Huang et al. (2026). This mixed-training strategy explicitly reinforces the agent’s fundamental deep research capabilities. The same data recipe is applied for both the 30B and 35B variants. Formally, given the mixed dataset $𝒟$, where each instance consists of a context $x$ (including the system prompt, visual inputs, and interaction history) and the target output sequence $y=\{y_1,\dots ,y_N\}$, the SFT objective is to minimize the standard auto-regressive negative log-likelihood:

${ℒ}_{\text{SFT}}=-{𝔼}_{(x,y)\sim 𝒟}\left[\sum _{i=1}^{\left|y\right|}\log {\pi }_{\theta }\left(y_i\mid x,y_{<i}\right)\right]$

(2)

where ${\pi }_{\theta }$ represents the policy of the base model and $y_{<i}$ denotes the preceding tokens.

RL

To push the agent beyond static SFT imitation and incentivize endogenous exploration, we employ Group Relative Policy Optimization (GRPO) Shao et al. (2024). GRPO computes advantages via intra-group relative rewards, efficiently eliminating the memory overhead of a separate value network. We construct a 2K moderate-difficulty RL dataset by executing four rollouts per trajectory (Sec.3.2) and strictly retaining instances with a Pass@4 score between 0 and 1. We apply a sparse binary reward, assigning $r=1$ for correct answers (judged by Qwen3-VL-30B-A3B-Instruct) and $r=0$ otherwise. To prevent formatting violations or repetitive loops from dominating the updates, we compute the corresponding advantage $\hat{A}$ but down-sample their negative gradients, applying them with only a 20% probability. The objective is defined as:

$\begin{array}{cc}{ℒ}_{\text{GRPO}}=& \frac{1}{G}\sum _{i=1}^G[\min (\frac{{\pi }_{\theta }\left(o_i\right)}{{\pi }_{\text{old}}\left(o_i\right)}{\hat{A}}_i,\\ & \text{clip}(\frac{{\pi }_{\theta }\left(o_i\right)}{{\pi }_{\text{old}}\left(o_i\right)},1-ϵ,1+ϵ){\hat{A}}_i)]-\beta 𝔻{}_{\text{KL}}\end{array}$

(3)

3.4 VideoDR-Bench

To establish our evaluation benchmark, we sample a subset from the rigorously filtered video pool. Since a robust benchmark strictly necessitates both answerability and high data fidelity, we introduce a scalable human-in-the-loop annotation framework to finalize the curation.

Specifically, given a video and an optional source URL, human annotators are instructed to pause at critical timestamps. They then utilize the Crop_Search tool to query salient visual entities, strictly verifying the consistency between the retrieved external evidence and the original frame. Based on these verified results, annotators formulate several seed VQA pairs per video. Subsequently, these seeds are fed into a multi-agent framework to synthesize complex, multi-hop reasoning questions. Specifically, the multi-agent pipeline operates as follows. First, a Drafting Agent brainstorms semantic directions to expand the seed VQA (e.g., expanding a base answer like “LeBron James” into related keywords such as his team, spouse, or MVP awards). These generated keywords are subsequently queried via a search engine. Next, a QA Generation Agent utilizes the retrieved web contexts to formulate novel multi-hop questions. To strictly enforce external tool dependency, we filter out any questions that the model can answer correctly without tool access (i.e., parametric knowledge leakage). Human annotators then manually verify the answerability of the remaining candidates based on the retrieved evidence. Following this, a Ranking Agent scores the validated questions, retaining only the highest-rated instance. Crucially, this top-ranked VQA can recursively serve as a new seed, enabling an iterative loop to synthesize increasingly complex, higher-hop reasoning tasks.

媒体内容 · 前往原文查看

Video Type	Duration (min)	Count	Percentage
Short	$\le 2$	92	46.0%
Medium	$2\sim 10$	68	34.0%
Long	$\ge 10$	40	20.0%

4 Experiments

4.1 Experimental Setups

We evaluate various Vision–Language Models (VLMs), including Gemini 2.5 Pro Comanici et al. (2025), GPT-5 OpenAI. (2025), Claude-4.5-Sonnet Anthropic (2025), Qwen3-VL-30B-A3B-Instruct Bai et al. (2025), Qwen3.5-35B-A3B Team (2026), Qwen3.5-397B-A3B Team (2026) and Kimi K2.5 Team et al. (2026), on VideoDR-Bench and VideoDR Liu et al. (2026). VideoDR-Bench is an evaluation benchmark comprising 100 human-annotated VQA pairs, designed to assess an agent’s complex reasoning capabilities by integrating video contexts with open-web exploration.

We evaluate the models under the Agentic setting: where the model is equipped with the full suite of visual and text tools as shown in Table 6.

Model	Video-DR	VideoDR-Bench (% Acc.)							Avg.
	Acc.	KNL	ENT	DLY	G&S	NWS	OTH	Overall
\rowcolorblue!6    Closed-Source Models
Gemini 2.5 Pro Comanici et al. (2025)	62.0	54.2	52.3	51.4	51.7	54.2	57.1	53.0	57.5
GPT-5 OpenAI. (2025)	57.0	50.8	45.5	48.6	48.3	45.8	42.9	48.0	52.5
Claude-4.5-Sonnet Anthropic (2025)	63.0	55.9	54.5	54.1	58.6	54.2	42.9	55.0	59.0
\rowcolorteal!6    Open-Source Models
Qwen3.5-397B-A13B Team (2026)	58.0	49.2	61.4	40.5	55.2	29.2	14.3	47.5	52.8
Kimi K2.5 Team et al. (2026)	61.0	54.2	52.3	51.4	51.7	54.2	57.1	53.0	57.0
Qwen3-VL-30B-A3B-Instruct Bai et al. (2025)	38.0	44.1	43.2	35.1	51.7	41.7	42.9	43.0	40.5
Video-DeepResearch-30B-A3B (Ours)	62.0	62.7	61.4	40.5	58.6	58.3	42.9	56.5	59.3
\rowcolorgreen!12 Improvement$\uparrow$	+24.0	+18.6	+18.2	+5.4	+6.9	+16.6	0.0	+13.5	+18.8
Qwen3.5-35B-A3B Team (2026)	42.0	45.8	50.0	40.5	44.8	33.3	28.6	43.5	42.8
Video-DeepResearch-35B-A3B (Ours)	68.0	66.1	65.9	56.8	62.1	41.7	42.9	60.0	64.0
\rowcolorgreen!12 Improvement$\uparrow$	+26.0	+20.3	+15.9	+16.3	+17.3	+8.4	+14.3	+16.5	+21.2

arxiv

Under identical interaction constraints for fairness, we extract the final prediction from the trajectory’s last step. Correctness is then evaluated by Qwen3-VL-30B-A3B-Instruct, adopting the official judge prompt from Tongyi DeepResearch Team et al. (2025).

4.2 Main Results

The main evaluation results are summarized in Table 4.1. Our proposed Video-DeepResearch demonstrates exceptional deep research capabilities across multiple model scales.

Video-DeepResearch-35B-A3B. Establishing a new state-of-the-art among all evaluated models with an overall average accuracy of 64.0%, our 35B variant surpasses the leading closed-source model, Claude-4.5-Sonnet (59.0%) by 5.0 percentage points. Compared to its foundation model (Qwen3.5-35B-A3B), Video-DeepResearch-35B demonstrates a remarkable +21.2% improvement, with particularly strong gains across knowledge-intensive (KNL: +20.3%) and entertainment (ENT: +15.9%) categories. Notably, Video-DeepResearch-35B achieves the highest score on VideoDR-Bench (65.4%) among all evaluated models.

Video-DeepResearch-30B-A3B. The 30B variant achieves 59.3% average accuracy, competitive with Claude-4.5-Sonnet (59.0%) and significantly eclipsing GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). Compared to its foundation model (Qwen3-VL-30B-A3B-Instruct), Video-DeepResearch-30B achieves substantial absolute improvements: a +24.0% surge on the VideoDR benchmark and a +13.5% gain on VideoDR-Bench. This explicit performance leap rigorously validates the efficacy of our trajectory synthesis and RL optimization pipeline even at compact scale.

Comparison with Proprietary Models. Despite their massive parameter counts and extensive training, proprietary models exhibit notable limitations on Video-DR tasks. GPT-5 (52.5%) lags significantly behind, likely due to insufficient optimization for visual tool usage. Gemini 2.5 Pro (57.5%) shows competitive performance but still falls short of both our variants. These results underscore that raw model scale alone does not guarantee effective Video-DR capability; specialized training pipelines are essential.

Fine-Grained Category Analysis. On the fine-grained level, our models exhibit distinct strengths across different video domains:

• Knowledge (KNL): Video-DeepResearch-35B achieves 66.1%, demonstrating superior capability in factual reasoning over video-grounded knowledge queries.

• Entertainment (ENT): Both variants excel (61.4% and 65.9% respectively), outperforming all proprietary models.

• Daily Life (DLY): Video-DeepResearch-35B significantly improves to 56.8% (+16.3% over base), indicating better generalization to common scenarios.

• News (NWS): Video-DeepResearch-30B shows particular robustness (58.3%), though Video-DeepResearch-35B (41.7%) suggests potential domain transfer challenges for the larger variant.

Scaling Insight. A critical observation emerges from our experiments: the improvement margins are not uniform across model sizes. While Video-DeepResearch-30B achieves +18.8% over its base, Video-DeepResearch-35B achieves +21.2%. This suggests that our training pipeline synergizes better with increased model capacity, particularly for complex multi-hop reasoning tasks. However, the relatively smaller improvement on News category (8.4% vs. 16.6% for 30B) indicates potential brittleness on temporally dynamic content that warrants further investigation.

4.3 Tool Usage Analysis

媒体内容 · 前往原文查看

Model	VideoDR		VideoDR-Bench
	Visual	Text	Visual	Text
Claude-4.5-Sonnet	1.83	3.38	2.25	3.24
GPT-5	0.00	0.12	0.31	1.43
Gemini-2.5-Pro	0.31	0.84	1.93	2.07
Qwen3.5-35B	0.04	0.58	0.20	0.70
Qwen3.5-397B	0.10	1.27	0.04	2.77
Base	1.82	2.21	1.75	2.40
Ours	2.33	4.24	2.98	3.81

To understand the behavioral patterns underlying our performance gains, we profile the tool invocation frequencies across different evaluation sets in Table 4.

Benchmark Characteristics. From the benchmark perspective, VideoDR-Bench rigorously compels models to execute significantly more visual and textual operations compared to VideoDR. For instance, GPT-5 increases visual tool calls from 0.00 to 0.31 and text calls from 0.12 to 1.43. This explicitly demonstrates that our benchmark circumvents parametric knowledge leakage prevalent in existing datasets, successfully enforcing genuine multi-step, tool-augmented open-web exploration.

Methodological Analysis. Our proposed Video-DeepResearch exhibits a profound shift in tool utilization compared to baseline models:

• Modality Bias Correction: Baseline Qwen3.5-397B executes only 0.10 visual operations per task while heavily favoring text tools (1.27). In contrast, Video-DeepResearch-30B achieves 2.33 visual and 4.24 text tool invocations on VideoDR, representing a fundamental restructuring of the agent’s exploration strategy.

• Balanced Multimodal Search: Driven by our carefully curated trajectory pipeline, Video-DeepResearch internalizes active spatiotemporal perception. Augmented by mixed-text training data, this leads to a highly balanced and exhaustive multimodal search strategy that dynamically overcomes the modality bias observed in baseline models.

• Scale-Appropriate Efficiency: Notably, Video-DeepResearch-30B achieves higher tool usage than even the 397B baseline, confirming that training methodology outweighs raw parameter count in determining agentic capability.

Key Insight. The strong correlation between tool usage diversity and task performance validates our core hypothesis: effective Video-DR agents must overcome the modality bias that causes models to rely on parametric knowledge. Our training pipeline successfully instills this capability, as evidenced by both the quantitative performance gains and the behavioral shift toward more exhaustive exploration.

4.4 Ablation Study

媒体内容 · 前往原文查看

Setting	VideoDR	VideoDR-Bench	Avg
Base	38.0	43.0	40.5
4k-SFT	44.0	48.0	46.0
7K-SFT	55.0	51.0	53.0
7K-SFT+7k-text-SFT	59.0	54.5	56.8
14k-SFT+2K-RL	62.0	56.5	59.3

To meticulously disentangle the contribution of each data curation and training phase, we conduct an ablation study on Video-DeepResearch-30B, as detailed in Table 5.

Video-Centric SFT (Base $\to$ 7K-SFT). Starting from the baseline (40.5%), introducing 4K synthesized Video-DR trajectories immediately yields a +5.5% average gain. Scaling this multimodal corpus to 7K further propels the performance to 53.0%. This explicit +12.5% trajectory-driven surge confirms that our visual grounding data is strictly necessary for the agent to internalize the foundational perception-exploration paradigm. Interestingly, the improvement is more pronounced on VideoDR-Bench (51.0% vs. 43.0% base) than on VideoDR (55.0% vs. 38.0% base), suggesting that visual grounding is particularly crucial for benchmark-quality evaluations.

Text-Augmented SFT (+7k-text-SFT). Subsequently, incorporating 7K text-only QA instances results in an additional +3.8% improvement (56.8%). This directly validates our hypothesis: explicitly injecting textual exploration data effectively mitigates the agent’s initial tool-invocation bias, thereby enforcing a more comprehensive open-web search capability. The cross-modal transfer effect suggests that textual deep research skills complement and enhance visual grounding behaviors.

RL Optimization (+2K-RL). Finally, applying GRPO on the 2K moderate-difficulty dataset achieves the peak overall accuracy of 59.3%. This final +2.5% performance leap underscores the necessity of reinforcement learning—pushing the model beyond static imitation to execute robust, self-driven exploration trajectories. The RL phase particularly benefits VideoDR-Bench performance (+2.0% to 56.5%), indicating that the learned exploration strategy generalizes well to benchmark-quality tasks.

Cumulative Design Insight. The incremental improvements from each training phase reveal an important principle: Video-DR capability emerges from the synergistic combination of visual grounding (trajectory SFT), textual deep research skills (mixed SFT), and autonomous exploration (RL). Neither component alone achieves optimal performance, and the full pipeline is essential for state-of-the-art results.

4.5 Discussion: Beyond Benchmarks

The experimental results prompt deeper reflections on the nature of video intelligence.

Emergence Over Scaling. Our findings challenge the prevailing assumption that sufficient model scale alone will yield emergent capabilities. The 397B parameter Qwen3.5-397B-A13B, despite its massive capacity, performs comparably to our 30B model trained with our specialized pipeline. This suggests that Video-DR capability is not merely a function of model size, but rather an emergent property that requires deliberate curriculum design—specifically, the decoupled perception-exploration paradigm we introduce. The agent must learn when to perceive and when to retrieve, a temporal coordination that cannot be extracted from static corpora alone.

Active Grounding as a Test of True Understanding. A provocative interpretation of our results concerns what tool usage actually measures. The parametric knowledge leakage observed in GPT-5 (achieving competitive accuracy with zero tool calls) reveals that benchmark performance alone can be decoupled from genuine video understanding. Our approach, by enforcing exhaustive visual grounding prior to web retrieval, effectively operationalizes a principle: understanding a video means being able to act upon it. The shift in tool invocation patterns (from 0.10 to 2.33 visual operations) thus represents not merely behavioral modification, but a fundamental restructuring of the agent’s epistemic strategy—from passive recall to active verification.

The Parity of Modalities. Perhaps the most counterintuitive finding is that training methodology can outweigh model scale by nearly an order of magnitude. Our 30B model achieves higher tool diversity than the 397B baseline, suggesting that modality bias is not an architectural limitation but a distributional artifact of training data. This implies that achieving true multimodal parity requires not just architectural unification, but data and training paradigm alignment—a more nuanced requirement than simply scaling model parameters.

5 Related Work

5.1 Video Understanding.

Early Video-LLMs typically rely on uniform frame sampling for single-turn inference Lin et al. (2024a, b); Zhang et al. (2024). Lacking dynamic visual querying mechanisms, they are prone to error accumulation and hallucinations. While recent agentic frameworks introduce interactive tools for active fine-grained perception Yang et al. (2025b); Zhang et al. (2025); Tian et al. (2025), they remain confined to closed-world video contexts. Consequently, they struggle with knowledge-intensive tasks that require external, verifiable evidence Wang et al. (2017); Fu et al. (2025b). To address this, we introduce Video-DeepResearch, a video deep research framework that couples internal video-grounding with iterative open-web exploration. By transitioning from closed-world parametric memory to an open-world collaborative verification loop, Video-DeepResearch enables highly robust, multi-source deep reasoning.

5.2 Multimodal-DeepResearch Systems.

Autonomous deep research agents have rapidly evolved from text-only systems Team et al. (2025); Wu et al. (2025a); Li et al. (2025b); Tao et al. (2025) to image-centric Vision-DR frameworks. Recent efforts employ reverse image search Geng et al. (2025), GRPO optimization Wu et al. (2025b); Shao et al. (2024), and entity-level cropping Narayan et al. (2025) to navigate static visual contexts. However, this trajectory entirely bypasses the continuous video modality. Unlike static images, Video-DR requires agents to decouple dense, spatiotemporal dynamics and conduct multi-step verification across noisy frames, presenting a distinctly more formidable challenge.

Concurrently, establishing rigorous evaluations for this new frontier remains elusive. While current benchmarks broadly assess static multimodal factuality Cheng et al. (2025b), external knowledge grounding Wang et al. (2017); Chen et al. (2023); Fu et al. (2025b), and image-based search workflows Jiang et al. ; Geng et al. (2025), they are inherently insufficient for video streams. Moreover, the scarce efforts to establish dedicated Video-DR benchmarks suffer from a critical bottleneck: a severe reliance on labor-intensive, unscalable manual annotation. To overcome this, we introduce a highly scalable, human-AI collaborative annotation framework for robust Video-DR evaluation.

6 Conclusion

We present Video-DeepResearch, the first unified framework for Video-DeepResearch that bridges scalable data synthesis, agent training, and rigorous evaluation. A preliminary study on existing agents reveals two critical failure modes: systematic visual tool aversion and parametric knowledge leakage, both of which undermine faithful assessment of Video-DR capabilities. Video-DeepResearch addresses these issues through a decoupled perception-exploration pipeline with stage-wise tool unlocking, producing 30K video-grounded QA pairs and 7K curated trajectories. Combined with a two-stage SFT–GRPO training recipe, our Video-DeepResearch-35B-A3B achieves 64.0% (SOTA), while the 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet. We further introduce VideoDR-Bench, a 200-instance multi-hop VQA benchmark where every question provably demands both visual search and external knowledge reasoning. We hope this work lays a solid foundation for next-generation video deep research agents.

Author Information

Full Author List: Zhen Fang¹, Yu Zeng¹, Wenxuan Huang, Yiming Zhao¹, Shiting Huang¹, Tianfei Ren¹, Qi Lu¹, Qingnan Ren¹, Qisheng Su¹, Lionel Z. Wang⁴, Qingyu Yin⁵, Shuang Chen⁶, Zehui Chen¹, Lin Chen¹, Zhenfei Yin⁷, Yao Hu², Shaohui Lin⁸, Wanli Ouyang³, Shaosheng Cao^2,9, Feng Zhao¹.

Affiliations: ¹USTC, ²Xiaohongshu Inc., ³CUHK, ⁴The Hong Kong Polytechnic University, ⁵ZJU, ⁶UCLA, ⁷Oxford, ⁸ECNU, ⁹THU.

The first authors with equal contribution are Zhen Fang, Yu Zeng, Wenxuan Huang, and Yiming Zhao. Yu Zeng and Wenxuan Huang serve as the project leaders. The corresponding authors are Wenxuan Huang, Shaosheng Cao and Feng Zhao.

Limitation

While Video-DeepResearch pioneers the first comprehensive pipeline integrating data construction and model training for the complex Video-DR task, this rigorous approach introduces certain trade-offs. Primarily, achieving our current level of performance incurs considerable computational overhead. To ensure high-quality data synthesis and robust model training, the framework demands substantial GPU resources, largely due to the concurrent requirements of large-scale model deployment and dynamic web search operations. Furthermore, to guarantee accurate and reliable evaluation, the construction of our benchmark currently relies on meticulous human annotation. While this ensures high fidelity of the evaluation standard, it restricts the rapid scalability of the dataset. In future work, we aim to mitigate these constraints by exploring computationally efficient pipelines, lightweight architectures, and automated LLM-based evaluation metrics to reduce human dependency.

References

• Anthropic (2025) Introducing claude sonnet 4.5. Note: https://www.anthropic.com/news/claude-sonnet-4-5 Cited by: §4.1, §4.1.
• K. Ataallah, E. M. Bakr, M. Ahmed, C. Gou, K. Pahwa, J. Ding, and M. Elhoseiny (2025) Infinibench: a benchmark for large multi-modal models in long-form movies and tv shows. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 19496–19523. Cited by: §3.1.
• S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §3.3, §4.1, §4.1.
• D. Ben-Ami, G. Serussi, K. Cohen, and C. Baskin (2025) HERBench: a benchmark for multi-evidence integration in video question answering. arXiv preprint arXiv:2512.14870. Cited by: §3.1.
• S. Chen, K. Feng, H. Chen, W. Huang, D. Dai, Q. Shou, Y. Lin, X. Yue, S. Gao, and T. Pang (2026) OpenSearch-vl: an open recipe for frontier multimodal search agents. arXiv preprint arXiv:2605.05185. Cited by: §1.
• Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M. Chang (2023) Can pre-trained vision and language models answer visual information-seeking questions?. arXiv preprint arXiv:2302.11713. Cited by: §5.2.
• J. Cheng, Y. Ge, T. Wang, Y. Ge, J. Liao, and Y. Shan (2025a) Video-holmes: can mllm think like holmes for complex video reasoning?. arXiv preprint arXiv:2505.21374. Cited by: §3.1.
• X. Cheng, W. Zhang, S. Zhang, J. Yang, X. Guan, X. Wu, X. Li, G. Zhang, J. Liu, Y. Mai, et al. (2025b) Simplevqa: multimodal factuality evaluation for multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4637–4646. Cited by: §5.2.
• G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1, §4.1, §4.1.
• K. Feng, M. Zhang, S. Chen, Y. Lin, K. Fan, Y. Jiang, H. Li, D. Zheng, C. Wang, and X. Yue (2026) Gen-searcher: reinforcing agentic search for image generation. arXiv preprint arXiv:2603.28767. Cited by: §1.
• C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. (2025a) Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24108–24118. Cited by: §3.1.
• M. Fu, Y. Peng, B. Liu, Y. Wan, and D. Chen (2025b) LiveVQA: live visual knowledge seeking. arXiv preprint arXiv:2504.05288. Cited by: §5.1, §5.2.
• X. Geng, P. Xia, Z. Zhang, X. Wang, Q. Wang, R. Ding, C. Wang, J. Wu, Y. Zhao, K. Li, et al. (2025) Webwatcher: breaking new frontier of vision-language deep research agent. arXiv preprint arXiv:2508.05748. Cited by: §1, §5.2, §5.2.
• A. Goel, S. Ghosh, V. Agarwal, N. Anand, K. Jayakumar, L. Koroshinadze, Y. Xu, K. Lyons, J. Case, K. Sapra, et al. (2026) MMOU: a massive multi-task omni understanding and reasoning benchmark for long and complex real-world videos. arXiv preprint arXiv:2603.14145. Cited by: §3.1.
• J. Hong, S. Yan, J. Cai, X. Jiang, Y. Hu, and W. Xie (2026) WorldSense: evaluating real-world omnimodal understanding for multimodal llms. External Links: 2502.04326, Link Cited by: §3.1.
• K. Hu, P. Wu, F. Pu, W. Xiao, Y. Zhang, X. Yue, B. Li, and Z. Liu (2025) Video-mmmu: evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826. Cited by: §3.1.
• W. Huang, Y. Zeng, Q. Wang, Z. Fang, S. Cao, Z. Chu, Q. Yin, S. Chen, Z. Yin, L. Chen, et al. (2026) Vision-deepresearch: incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601.22060. Cited by: §1, §1, §3.3.
• [18] D. Jiang, R. Zhang, Z. Guo, Y. Wu, P. Qiu, P. Lu, Z. Chen, G. Song, P. Gao, Y. Liu, et al. Mmsearch: unveiling the potential of large models as multi-modal search engines. In The Thirteenth International Conference on Learning Representations, Cited by: §5.2.
• C. Li, Y. Chen, Y. Ji, J. Xu, Z. Cui, S. Li, Y. Zhang, W. Wang, Z. Song, D. Zhang, et al. (2025a) Omnivideobench: towards audio-visual understanding evaluation for omni mllms. arXiv preprint arXiv:2510.10689. Cited by: §3.1.
• K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. (2025b) WebSailor: navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592. Cited by: §1, §5.2.
• K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. (2024) Mvbench: a comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195–22206. Cited by: §3.1.
• Z. Li, C. Li, X. Mao, S. Lin, M. Li, S. Zhao, Z. Xu, X. Li, Y. Feng, J. Sun, et al. (2026) Sekai: a video dataset towards world exploration. Advances in Neural Information Processing Systems 38. Cited by: §3.1.
• B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan (2024a) Video-llava: learning united visual representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 5971–5984. Cited by: §5.1.
• J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han (2024b) Vila: on pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26689–26699. Cited by: §5.1.
• C. Liu, X. Yu, Z. Chang, Z. Huang, S. Zhang, H. Lian, K. Wang, R. Xu, S. Hu, J. Hou, et al. (2026) Watching, reasoning, and searching: a video deep research benchmark on open web for agentic video reasoning. arXiv preprint arXiv:2601.06943. Cited by: §1, §2, Table 1, §4.1.
• X. Ma, H. Qiu, G. Zhang, Z. Zeng, S. Yang, L. Ma, and F. Zhao (2025) Stage: stable and generalizable grpo for autoregressive image generation. arXiv preprint arXiv:2509.25027. Cited by: §1.
• R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. (2021) Webgpt: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Cited by: §1.
• K. Narayan, Y. Xu, T. Cao, K. Nerella, V. M. Patel, N. Shiee, P. Grasch, C. Jia, Y. Yang, and Z. Gan (2025) Deepmmsearch-r1: empowering multimodal llms in multimodal web search. arXiv preprint arXiv:2510.12801. Cited by: §5.2.
• OpenAI. (2025) Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: §4.1, §4.1.
• A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: Appendix C.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §3.3, §5.2.
• K. Tao, Y. Zheng, J. Xu, W. Du, K. Shao, H. Wang, X. Chen, X. Jin, J. Zhu, B. Yu, et al. (2026) Lvomnibench: pioneering long audio-video understanding evaluation for omnimodal llms. arXiv preprint arXiv:2603.19217. Cited by: §3.1.
• Z. Tao, J. Wu, W. Yin, J. Zhang, B. Li, H. Shen, K. Li, L. Zhang, X. Wang, Y. Jiang, et al. (2025) Webshaper: agentically data synthesizing via information-seeking formalization. arXiv preprint arXiv:2507.15061. Cited by: §1, §5.2.
• K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. (2026) Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: §4.1, §4.1.
• Q. Team (2026) Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804. Cited by: §1, §3.1, §3.1, §3.3, §4.1, §4.1, §4.1.
• T. D. Team, B. Li, B. Zhang, D. Zhang, F. Huang, G. Li, G. Chen, H. Yin, J. Wu, J. Zhou, et al. (2025) Tongyi deepresearch technical report. arXiv preprint arXiv:2510.24701. Cited by: §4.1, §5.2.
• S. Tian, R. Wang, H. Guo, P. Wu, Y. Dong, X. Wang, J. Yang, H. Zhang, H. Zhu, and Z. Liu (2025) Ego-r1: chain-of-tool-thought for ultra-long egocentric video reasoning. arXiv preprint arXiv:2506.13654. Cited by: §5.1.
• P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Hengel (2017) Fvqa: fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence 40 (10), pp. 2413–2427. Cited by: §5.1, §5.2.
• W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, M. Ding, X. Gu, S. Huang, B. Xu, et al. (2025) Lvbench: an extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22958–22967. Cited by: §3.1.
• W. Wang and Y. Yang (2026) Videoufo: a million-scale user-focused dataset for text-to-video generation. Advances in Neural Information Processing Systems 38. Cited by: §3.1.
• H. Wu, D. Li, B. Chen, and J. Li (2024) Longvideobench: a benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems 37, pp. 28828–28857. Cited by: §3.1.
• J. Wu, B. Li, R. Fang, W. Yin, L. Zhang, Z. Tao, D. Zhang, Z. Xi, G. Fu, Y. Jiang, et al. (2025a) Webdancer: towards autonomous information seeking agency. arXiv preprint arXiv:2505.22648. Cited by: §1, §5.2.
• J. Wu, Z. Deng, W. Li, Y. Liu, B. You, B. Li, Z. Ma, and Z. Liu (2025b) MMSearch-r1: incentivizing lmms to search. arXiv preprint arXiv:2506.20670. Cited by: §1, §5.2.
• H. Yang, S. Yue, and Y. He (2023) Auto-gpt for online decision making: benchmarks and additional opinions. arXiv preprint arXiv:2306.02224. Cited by: §1.
• J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie (2025a) Thinking in space: how multimodal large language models see, remember, and recall spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10632–10643. Cited by: §3.1.
• Z. Yang, S. Wang, K. Zhang, K. Wu, S. Leng, Y. Zhang, B. Li, C. Qin, S. Lu, X. Li, et al. (2025b) Longvt: incentivizing" thinking with long videos" via native tool calling. arXiv preprint arXiv:2511.20785. Cited by: §3.1, §5.1.
• H. Zhang, X. Gu, J. Li, C. Ma, S. Bai, C. Zhang, B. Zhang, Z. Zhou, D. He, and Y. Tang (2025) Thinking with videos: multimodal tool-augmented reinforcement learning for long video reasoning. arXiv preprint arXiv:2508.04416. Cited by: §5.1.
• Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li (2024) Llava-video: video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. Cited by: §5.1.

Appendix A Appendix

Appendix B Training Details

Training Setup and Hyperparameters. The supervised fine-tuning (SFT) was conducted on a high-performance compute cluster comprising $4$ nodes, each equipped with $8\times 80$ GB GPUs ($32$ GPUs in total), utilizing the Megatron-LM framework. To accommodate the ultra-long context length of $80,000$ tokens without data packing, we employed a highly optimized mixed-parallelism strategy. Specifically, we configured a Tensor Parallelism (TP) size of $4$, a Context Parallelism (CP) size of $2$, and an Expert Parallelism (EP) size of $8$. Sequence Parallelism (SP) was also enabled to further reduce the memory footprint. The training spanned $3$ epochs with a micro-batch size of $1$ and a global batch size of $64$. The learning rate was governed by a linear warmup and decay schedule, warming up over the first $5\%$ of training steps to a peak of $1\times {10}^{-5}$, and subsequently decaying to a minimum of $5\times {10}^{-7}$.

Optimization and Efficiency Enhancements. To ensure efficient training of the Mixture-of-Experts (MoE) architecture, we applied an auxiliary loss coefficient of $1\times {10}^{-6}$ for load balancing and set the expert capacity factor to $2.0$ to mitigate token dropping. Advanced MoE computational optimizations were integrated, including permute operation fusion, Grouped GEMM, and the overlapping of shared-expert computation with communication. Furthermore, strict memory optimization techniques were adopted to prevent Out-of-Memory (OOM) errors during long-context training. We utilized FlashAttention as the primary attention backend and PyTorch’s expandable segments feature (expandable_segments:True) to minimize memory fragmentation. Full activation checkpointing was applied uniformly at every single layer, trading computation for memory. For system-level efficiency, fused cross-entropy loss was enabled, CPU threading was optimized with $32$ OpenMP threads, and the data processing pipeline was heavily parallelized using $128$ preprocessing processes and $8$ DataLoader workers. Checkpoints were serialized in the Safetensors format every $500$ steps, deliberately omitting optimizer and random number generator (RNG) states to conserve storage overhead.

Appendix C Data Details

For keyframe extraction, we compute inter-frame similarities using CLIP-ViT-L/14@336px Radford et al. (2021). To reduce redundancy, we discard consecutive frames with a similarity score exceeding 0.8, alongside any uninformative monochromatic frames. The maximum number of keyframes per video is strictly capped at 20. Notably, this preprocessing configuration is uniformly applied across both the data synthesis pipeline and all evaluation phases, including the processing of VideoDR.

Appendix D Annotation Details

The construction of VideoHunt involves a rigorous human annotation process to ensure benchmark quality and reliability. The entire annotation effort spans approximately three weeks.

Annotator Team. We recruit a team of 8 annotators, all with prior professional experience in multimodal large language model data annotation. Each annotator possesses strong familiarity with video understanding, visual search, and multi-hop reasoning tasks.

Quality Assurance. Before the formal annotation phase, all annotators undergo a structured training session covering task definitions, tool usage (e.g., Crop_Search), and common pitfalls such as parametric knowledge leakage. This is followed by a qualification test on a held-out pilot set; only annotators meeting a predefined accuracy threshold are admitted to the main annotation. During production, the workflow is organized into two decoupled stages: an annotation stage, where annotators create and verify VQA instances, and a quality inspection stage, where a separate group of reviewers cross-checks each instance for answerability, visual groundability, and factual correctness. Instances flagged during inspection are returned for revision or discarded.

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

媒体内容 · 前往原文查看

Parameter	Type	Req.	Description
select_crop_search — pick frames, crop a bounding box from each, run a reverse-image / visual web search per crop.
selections	list of objects (1–8)	✓	A batch of (frame, bbox) selections to crop and search in a single call. Use several selections to cast a wide net, or a single tight crop to zoom in on one discriminative region.
.frame_index	integer ($\ge 0$)	✓	0-based index of the frame in the video’s frame list.
.bbox	list[float], length 4	✓	Crop region in normalised $[x_1,y_1,x_2,y_2]$ coordinates with $0.0\le x_1<x_2\le 1.0$ and $0.0\le y_1<y_2\le 1.0$. Use $[0,0,1,1]$ for the full frame.
goal	string	✓	Precise statement of what to identify and the fact to look up about it.
Returns: for each crop, the top web results (title, URL, snippet) and any recognised entities (people, products, logos, landmarks).
search — run a web search and return the top results from Google / Serper / Zhipu (with automatic fallback).
query	string or list[string]	✓	A single search query or a list of queries run in parallel. Prefer several short, focused queries over one long compound query.
num_results	integer	—	Max results returned per query (default 10).
Returns: per query, an ordered list of $\{\text{title},\text{url},\text{snippet}\}$ entries.
visit — fetch one or more web pages and have a summary model distil them into a goal-conditioned JSON.
url	string or list[string]	✓	The URL to fetch, or a list of URLs to fetch in parallel. Typically a URL returned by an earlier search call.
goal	string	✓	Precise statement of what to look for on the page; the summary model uses it to decide what to keep vs. discard.
Returns: per URL, a JSON object with rational (why the page is relevant), evidence (verbatim supporting spans), and summary (one-paragraph synthesis). Raw HTML is not exposed to the agent.