K. Steunou is with the SAMOVAR Laboratory, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France, and also with Moments Lab, Paris, France (e-mail: killian.steunou@ip-paris.fr).

Y. Tevissen is with Moments Lab, Paris, France (e-mail: yannis.tevissen@momentslab.com).

M. A. El Yacoubi is with the SAMOVAR Laboratory, Télécom SudParis, Institut Polytechnique de Paris, Palaiseau, France (e-mail: mounim.el_yacoubi@telecom-sudparis.eu).

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Abstract

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy–cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.

Index Terms: 

efficient video understanding, VideoLLMs, multimodal large language models, computational efficiency

I Introduction

媒体内容 · 前往原文查看

Video content spans short social media clips, instructional videos, movies and long-form egocentric recordings, combining spatial, temporal and multimodal cues: frames, audio, speech, subtitles and overlaid text. Video understanding has shifted from task-specific architectures to large pre-trained foundation models, trained on video-text corpora to support captioning, question answering, retrieval, spatiotemporal grounding and dense summarization [1, 2, 3, 4, 5]. Video large language models (VideoLLMs) extend text-only LLMs with visual encoders and, in audiovisual systems, audio encoders [2, 6], supporting open-ended reasoning and instruction following on video-centric tasks, often without task-specific fine-tuning.

However, these advances come with substantial computational and memory costs [7]: video encoders may process hundreds of high-resolution frames per clip across multiple modalities and long temporal contexts, and large language backbones add attention compute and inference-time memory overhead [8, 9]. Efficient VideoLLMs aim to retain these semantic and reasoning capabilities while reducing parameter count, floating-point operations (FLOPs) per input, latency or memory.

Typical VideoLLMs share a pipeline made of four stages: (1) construct the visual input by selecting frames, patches and resolution, (2) encode it with a vision backbone, (3) reduce and map the encoded representations to the LLM input space, and (4) process them together with a textual prompt in the LLM.

Recent work explores the capability vs. efficiency trade-off throughout the pipeline: selecting fewer frames before encoding, lighter vision backbones, compressing connector outputs, pruning visual tokens inside the LLM, and reducing the visual key-value (KV) cache [10, 11, 12, 13, 14]. Audiovisual systems additionally compress audio tokens or use sound to guide visual selection [15, 16]. Figure 1 traces these mechanisms across the four pipeline stages. Because these ideas are often proposed in isolation, tied to particular tasks such as captioning, question answering (QA) or temporal localization, and evaluated with heterogeneous metrics, it is difficult to determine where computational cost actually goes and which strategy is most effective under a given constraint. This fragmentation motivates a video-specific synthesis that relates reported efficiency gains to their pipeline stage, input coverage and evaluation conditions.

Recent surveys approach video understanding from complementary perspectives. Madan et al. [3], Nguyen et al. [4], and Tang et al. [2] review video foundation models, video-language learning, and VideoLLM architectures, respectively, emphasizing capabilities, tasks and benchmarks. Other surveys focus on long video understanding [17], temporal grounding [18], evaluation protocols [19], and omni-modal language models [20]. General multimodal LLM (MLLM) surveys place video within a broader landscape of modalities and architectures [6, 21, 22].

Efficiency-focused surveys overlap more directly with our scope. Jin et al. [23] cover efficient MLLM architectures, vision and language components, and training strategies, with video discussed as an application. Shao et al. [24] organize token compression by its underlying mechanisms across images, videos and audio; they also compare video compression methods under specified host models and token budgets. Their treatment provides a mechanism-centered account of token reduction, while our scope additionally includes frame-selection strategies and efficient video-encoder architectures, including mechanisms evaluated before the emergence of VideoLLMs.

Two recent surveys explicitly adopt a pipeline perspective. Zhang et al. [25] organize Large Vision-Language Models (LVLM) inference around encoding, prefilling and decoding, including keyframe selection, and analyze how optimization at one stage affects downstream bottlenecks. Wu et al. [26] organize MLLM compression by input, encoder, projector and LLM intervention points, crossed with five compression operations. These works establish pipeline structure and cross-stage cost interactions as shared foundations for efficiency analysis.

Our contribution is a video-centric synthesis built on these foundations. We connect frame selection and video-encoder design to connector compression and LLM-side inference, and examine how audio-token reduction and audio-guided visual selection affect the joint audiovisual workload. We assemble literature-reported comparisons within shared host models, input settings and token budgets wherever available, and distinguish these from comparisons across heterogeneous systems. Our emphasis is on how temporal coverage, encoder cost and multimodal token budgets jointly determine the benefits and limits of video inference-efficiency mechanisms.

In our survey, a VideoLLM is an encoder–connector–LLM system (illustrated in Figure 2) that provides video representations and a textual prompt to a pretrained LLM; visual-only systems encode frames, while audiovisual VideoLLMs additionally encode synchronized audio (Section II-A details how the surveyed methods were selected). We use “efficient” in a system-level sense: for a given task and hardware regime, an efficient method preserves or improves semantic performance while reducing parameter count, FLOPs per input, wall-clock latency, or memory; power and energy are also relevant but remain rarely reported [19]. Sections III-B and IV make this definition concrete through pipeline costs and the metrics reported in the literature.

• We synthesize video-specific inference-efficiency mechanisms across frame sampling, vision-encoder design, connector-level reduction and LLM-side processing, connecting upstream temporal coverage and encoding cost to downstream token and memory budgets.

• We assemble literature-reported accuracy–cost comparisons and identify which methods can be compared under shared hosts and evaluation settings. We separate these comparisons from heterogeneous cross-paper results and make differences in input protocols and FLOP-accounting boundaries explicit.

• We examine audiovisual efficiency through audio-token compression, audio-guided visual selection and joint token budgets, and use the evidence across stages to identify evaluation gaps and priorities for efficient VideoLLMs.

The remainder of this survey is structured as follows. Section II presents our paper-selection protocol, and defines the tasks and evaluation protocols; Section III reviews representative VideoLLM architectures and their computational bottlenecks; Section IV introduces the taxonomy and compares methods on shared benchmarks; Sections V and VI discuss trends and open challenges, and conclude.

II Preliminaries: Survey Scope, Tasks and Evaluation Protocols

II-A Survey Scope and Paper Selection

We survey efficiency mechanisms along the inference pipeline, from frame selection and modality encoding to connector-level token reduction, LLM prefilling (the forward pass over the full prompt, before any token is generated), decoding, and KV-cache use. We identified candidate methods through keyword searches on arXiv and Google Scholar, combining VideoLLM terms with efficiency terms such as token pruning, token merging, frame selection and KV-cache compression, and through backward and forward citation snowballing from the surveys discussed in the introduction and from each retained method. We cover papers published or posted as preprints up to August 2026.

A method enters the taxonomy when it contributes or evaluates a targeted mechanism and reports a concrete effect on parameter count, FLOPs, retained-token count, latency, or memory. We focus on VideoLLMs developed since late 2022. Earlier frame-sampling and vision-encoder methods are included when they remain components or direct antecedents of current pipelines. Audiovisual methods are included when they reduce the audio-token stream, use audio to reduce visual processing, or bound the joint audiovisual token stream. Training-only methods, generic LLM optimizations, and image-only techniques are cited as adjacent context when they establish or directly supply a mechanism adopted by VideoLLMs. The taxonomy has no model-size limit, but our quantitative tables emphasize language backbones around 7B–8B parameters, so a mechanism demonstrated only on a larger host appears in the taxonomy but not in the comparisons. Section IV-A details the model-size and reporting conventions behind our comparisons.

These searches surfaced several hundred candidate papers. We screened titles and abstracts against the criteria above, and read the remaining papers in full, retaining 125. Figure 4 shows all of them, marking the encoder and sampling methods that predate VideoLLMs. A paper appears in several families when it reduces cost at several stages, so family sizes add up to more than the number of papers. The selection is representative: new efficiency methods appear every month, and many recent methods apply an established lever at a different stage or granularity. When several papers instantiate the same mechanism, we keep those with the most complete efficiency reporting and the clearest evaluation protocol, cite close variants as context, and favor methods whose input and measurement settings support the controlled comparisons of Section IV.

II-B Tasks, Benchmarks and Evaluation

Video understanding spans classification, grounding, captioning, retrieval, question answering (QA) and dialogue, operating on RGB (Red Green Blue) frames with optional synchronized audio and derived text such as subtitles, automatic speech recognition (ASR) transcripts or optical character recognition (OCR) tokens. Each task family has standard datasets and metrics: action recognition and temporal localization (top-1/top-5 accuracy; mAP at temporal IoU thresholds) on Kinetics [27], Something-Something V2 [28] and Ego4D [29]; clip-level and dense captioning (BLEU, METEOR, ROUGE-L, CIDEr) on MSR-VTT [30] and ActivityNet Captions [31]; video QA (accuracy) on ActivityNet-QA [32], NExT-QA [33] and EgoSchema [34]; text–video retrieval (R@K, median rank) on caption datasets and narrated corpora such as HowTo100M [35]; and temporal grounding (R@K at temporal IoU) on Charades-STA [36] and Ego4D NLQ [29]. Efficiency-specific protocols are discussed in Section IV-A.

VideoLLM benchmarks complement these task-specific datasets by evaluating multiple capabilities under standardized protocols, most commonly through multiple-choice or structured QA, with emphasis on temporal reasoning beyond single-frame cues, long-context comprehension, and modality ablations. The efficiency comparisons later in this survey concentrate on MVBench [37], Video-MME [38], EgoSchema [34] and LongVideoBench [39] because they are the benchmarks most often shared by the methods we survey (Tables II–VII). Table I summarizes the benchmarks that appear in our comparisons and discussion; a full inventory of recent VideoLLM benchmarks is provided in the supplementary material.

媒体内容 · 前往原文查看

Benchmark	Mod.	Fmt.	Dur.	#V	#Q
MVBench [37]	V	MCQ	S	3,641	4,000
Video-MME [38]	V+A+T	MCQ	S/M/L	900	2,700
EgoSchema [34]	V	MCQ	M	5,063	5,063
LongVideoBench [39]	V+T	MCQ	M/L	3,763	6,678
MLVU [40]	V	MCQ	M/L	1,730	3,102
RVS-Ego / RVS-Movie [41]	V	OE	L	32	3,500

III VideoLLM Architectures and Computational Bottlenecks

We first review representative VideoLLMs, grouped by four families (short-video chat systems, unified image-video models, long-video and streaming systems, and audiovisual models) which determine where tokens are produced and how many. We then formalize the compute and memory costs of the resulting encoder–connector–LLM pipeline, which Section IV uses as its common basis for comparison.

III-A Representative VideoLLM Architectures

Tang et al. [2] distinguish three VideoLLM families by how video information reaches the LLM: Video Analyzer $\times$ LLM systems convert the video into textual evidence (captions, timestamped events, serialized object tracks, ASR or OCR) before LLM processing; Video Embedder $\times$ LLM systems map continuous encoder representations into the LLM input space through a connector; and hybrid (Analyzer + Embedder) $\times$ LLM systems provide both. We restrict this survey to the Embedder family (the largest, comprising 79 of the 127 systems Tang et al. catalog) because its encoder–connector–LLM structure matches the system boundary of our efficiency analysis: frame sampling, encoder cost, connector compression, multimodal token counts, LLM prefilling and KV-cache behavior. Analyzer-centric and hybrid systems would require accounting for the upstream expert models that produce textual analyses, and fall outside this pipeline-based scope. Figure 2 summarizes this framework.

Prompting lets the same backbone serve captioning, question answering, retrieval, temporal grounding and summarization without task-specific heads, so pipeline-level efficiency gains apply across all of them. Within this template, the most representative VideoLLMs differ mainly in their choice of encoders, connectors and language backbones, their target video length, and whether they use audio.

Short-video VideoLLMs and chat-centric systems. A first generation of VideoLLMs extends image-based VLMs (Vision Language Models) to short clips. Video-LLaMA [42] establishes the canonical pattern: CLIP [43] or ViT [44] vision encoders, ImageBind audio features [45], and a Q-Former connector [46] mapping both streams into Vicuna tokens [47]. VideoChat [48] and Valley [49] add chat-centric instruction tuning, Video-ChatGPT [50] popularizes GPT-based self-instruct training data, and mPLUG/mPLUG-2 [51, 52] apply dual-encoder contrastive pretraining to short video QA.

Unified image-video LLMs. A second wave moves to unified image-video models reusing image encoders with sparse frame sampling. The LLaVA family [53, 54] adds temporal pooling, LLaMA-VID [55] compresses each frame to two visual tokens, making long VideoQA feasible, and MiniGPT4-Video [56] interleaves visual and textual tokens, later serving as the backbone of Goldfish [57]. General-purpose VLMs such as Qwen2-VL [58] and InternVL [59] adopt the same unified pipeline, and InternVideo2.x [60, 61] shows that high-capacity video encoders with lightweight connectors compete favorably on MVBench [37] and Video-MME [38].

Long-video and streaming VideoLLMs. As long-video benchmarks emerged (EgoSchema [34], LongVideoBench [39], TVQA-long [57]), a third line targeted minute-to-hour contexts under strict limits: hierarchical memory approaches (MovieChat [62], LongVLM [7], MA-LMM [63]) compress visual tokens into multi-scale representations or explicit memory modules; streaming and retrieval methods (VideoStreaming [64], VideoLLM-online [65], VideoLLM-MoD [66], Goldfish [57]) maintain constant token budgets; $\infty$-Video [67] adds training-free long-term memory and frame selection around existing VideoLLMs [42, 37]; and the VideoChat family refines temporal encoding, reinforcement tuning for grounding, and multi-agent planning (VideoChat-T [68], VideoChat-R1 [69], VideoChat-M1 [70]).

Audiovisual VideoLLMs. Audiovisual VideoLLMs keep the same encoder–connector–LLM template while adding synchronized audio. Video-LLaMA maps ImageBind audio and ViT video features into Vicuna through separate Q-Formers; VideoLLaMA 2 [71] replaces this interface with spatial-temporal convolution connectors; and recent systems such as Qwen2.5-Omni [72] and OmniVinci [73] use dedicated visual and audio encoders with learned temporal alignment before a shared language core. These architectures add an audiovisual dimension to the taxonomy: audio adds an encoder and token stream, but the efficiency question remains how much encoded evidence reaches the LLM and at what cost.

III-B Sources of Computational Cost and Architectural Bottlenecks

We now formalize the dominant compute and memory scaling factors of the encoder–connector–LLM pipeline. Frame count and resolution determine encoder cost and the number of modality tokens produced; connector compression controls how many of those tokens enter the LLM; and the resulting context length determines LLM prefilling cost and KV-cache memory during decoding.

媒体内容 · 前往原文查看

We denote by $T$ the number of video frames fed to the encoder, by $H\times W$ the spatial resolution of each frame, and by $P\times P$ the patch size used by a frame-wise ViT encoder. The number of spatial patches per frame is $N_p=\frac{H}{P}\cdot \frac{W}{P},$ so the encoder initially produces $N_v^{\mathrm{enc}}=T\cdot N_p=T\cdot \frac{H}{P}\cdot \frac{W}{P}$ (up to special tokens). For video transformers using temporal tubelets of length $\tau$, $T$ is replaced by $T/\tau$. We write $N_a^{\mathrm{enc}}$ for the audio-encoder output length and ${\tilde{N}}_v,{\tilde{N}}_a$ for the visual and audio token counts retained after connector-side pooling, projection or resampling. With $N_t$ text tokens (prompt, history and any previously generated tokens), the LLM context length is $L=N_t+{\tilde{N}}_v+{\tilde{N}}_a.$ For a joint Q-Former that replaces both modality streams with $N_q$ query outputs, the corresponding context is $L=N_t+N_q$. A transformer block’s hidden width is its per-token embedding dimension: $d_v$ for the video encoder, $d_a$ for the audio encoder and $d$ for the LLM; $d_{\mathrm{ff}}$ and $d_{\mathrm{ff},a}$ are the corresponding feed-forward widths, and $d_{\mathrm{KV}}$ the total key/value width stored per token. The generic transformer-layer expressions below use $N$ and $d$ for the token count and width of the block in question. Figure 3 summarizes these bottlenecks visually.

Video and modality encoders. For a fixed 2D CNN (Convolutional Neural Network) applied frame-wise, encoder cost scales as ${\text{FLOPs}}_{\text{enc}}^{\text{2D}}=T\cdot C_{\mathrm{frame}}(H,W),$ where $C_{\mathrm{frame}}(H,W)$ is the cost of one pass through the chosen backbone; thus cost is linear in $T$ at fixed resolution and architecture. 3D CNNs and video transformers add temporal interactions. For a transformer layer processing a sequence of $N$ tokens, the attention and MLP (Multi-Layer Perceptron) costs scale as ${\text{FLOPs}}_{\text{attn}}\propto N\cdot d^2+N^2\cdot d,{\text{FLOPs}}_{\text{mlp}}\propto N\cdot d\cdot d_{\mathrm{ff}}.$

For a frame-wise ViT, the attention-mixing term summed across frames is $T{N}_p^{2}d$; only full joint space–time attention incurs ${\left(N_v^{\mathrm{enc}}\right)}^{2}d$, while factorized architectures lie between these regimes. Increasing the frame count or spatial resolution nevertheless inflates encoder cost. Long-video VideoLLMs often process hundreds of frames or minute-long clips via sliding windows or dense sampling, so the encoder alone can dominate total cost unless frames are subsampled or pooled.

Audio encoders usually begin from a denser temporal signal than sparsely sampled video, but their output length and cost depend strongly on convolutional stride, pooling and architecture. For a transformer layer operating on $N_a^{\mathrm{enc}}$ audio tokens, ${\text{FLOPs}}_{\text{audio,attn}}\propto N_a^{\mathrm{enc}}\cdot d_a^2+{\left(N_a^{\mathrm{enc}}\right)}^2\cdot d_a,$ with a further $N_a^{\mathrm{enc}}{d}_a{d}_{\mathrm{ff},a}$ contribution from the MLP; convolutional front ends have architecture-specific costs. Audio may be negligible after aggressive downsampling or material in long-form audiovisual inputs; it cannot be ranked against the visual stream from sampling rates alone because each video frame produces many spatial patch tokens. Additional ASR, OCR or subtitle-processing modules likewise add costs that should be reported separately [4, 17].

Connectors and cross-modal fusion. Connectors project high-dimensional spatiotemporal features (visual and audio tokens) into the LLM token space. In the simplest case, visual and audio tokens are flattened and passed through linear layers or small MLPs, yielding a cost ${\text{FLOPs}}_{\text{proj}}\propto N_v^{\mathrm{enc}}{d}_{v}d+N_a^{\mathrm{enc}}{d}_{a}d.$ More sophisticated connectors, such as Q-Former [46] or cross-attention modules, use a set of $N_q$ learnable query tokens attending over $N_s=N_v^{\mathrm{enc}}+N_a^{\mathrm{enc}}$ source tokens. Including query, key, value and output projections, the cross-attention cost per layer scales as ${\text{FLOPs}}_{\text{cross-attn}}\propto \left(N_q+N_s\right)d^2+N_q{N}_{s}d.$ Although $N_q$ is usually small, the source sequence $N_s$ can still be large. Many VideoLLMs therefore apply temporal or spatial pooling, audio downsampling, or selective token fusion before cross-attention, often enforcing a fixed joint token budget [2, 20].

LLM context length and KV cache. Once projected, the retained visual and audio tokens are concatenated (or interleaved) with textual tokens and processed by the LLM. In a standard transformer layer, self-attention over $L=N_t+{\tilde{N}}_v+{\tilde{N}}_a$ tokens has cost ${\text{FLOPs}}_{\text{LLM,attn}}\propto L{d}^2+L^{2}d,$ while feed-forward blocks contribute ${\text{FLOPs}}_{\text{LLM,mlp}}\propto Ld{d}_{\mathrm{ff}}.$ Whether the quadratic attention term or the linear feed-forward term dominates depends on the hidden width and the retained token counts. Among those tokens, visual patches usually outnumber the rest before compression, while audio can become substantial in long audiovisual inputs. Autoregressive decoding also stores key-value (KV) caches for each layer, with memory scaling ${\text{Mem}}_{\text{KV}}\propto 2B\cdot n_{\mathrm{layers}}\cdot L\cdot d_{\mathrm{KV}}\cdot b,$ where $B$ is the batch size, $n_{\mathrm{layers}}$ the number of decoder layers and $b$ the bytes per stored element; grouped- and multi-query attention reduce $d_{\mathrm{KV}}$. This limits feasible context length for multi-turn dialogue grounded in long videos, especially when audio, OCR or subtitles share the same context window [17, 74].

The dominant regime also changes between prefilling and autoregressive decoding. Attention-score computation during prefilling is quadratic in $L$, although kernels, hardware and the linear-in-$L$ projection and feed-forward terms determine whether execution is actually compute-bound. At each decoding step, attention mixing over the cached prefix costs $O\left(Ld\right)$ per layer, alongside $O\left(d^2\right)$ projection and feed-forward work, and is often constrained by memory traffic [75]. Reducing input tokens therefore benefits both stages, while KV-cache compression primarily targets decoding memory; their relative impact varies across interactive, batch and offline workloads.

Because the relative importance of encoding, connector token count and LLM prefill, decoding and cache growth is architecture- and workload-dependent, the next section organizes methods by the pipeline stage at which they reduce cost.

IV Taxonomy of Efficiency Mechanisms in VideoLLMs

媒体内容 · 前往原文查看

We analyze efficiency using the encoder–connector–LLM decomposition introduced in Section III. After defining the reporting conventions used in this survey, we organize mechanisms by the pipeline stage at which they act: input construction and selection, encoder computation, encoded representations and connector, and LLM execution and state. Figure 4 summarizes the taxonomy; audiovisual methods are included when audio compression or audio-guided selection directly reduces VideoLLM inference cost.

IV-A Comparison Protocol

As reviewed in Section III-B, VideoLLM papers mix analytical indicators and system-level measurements under heterogeneous assumptions about video length, resolution, modality coverage and hardware; we compare results only when their measurement scope and input protocol are explicit. Analytical, hardware-independent indicators include parameter count, FLOPs per input and retained-token count or ratio (we reserve FLOP/s for rate-based throughput). Their scope depends on the pipeline stage: vision-encoder comparisons report encoder size and GFLOPs together with the clip configuration [125, 114], whereas connector and token-reduction comparisons report host-LLM size, and accounting boundaries differ even then: HoliTom [142] and HieraVid [141] report LLM prefilling FLOPs while EarlyTom includes vision-encoder FLOPs [126]. For audiovisual systems, whose video and audio encoders can have very different profiles and whose connectors range from linear projections to Q-Formers [46] or Perceiver resamplers [165], we distinguish encoder, connector and LLM costs whenever the source provides them [23].

Analytical indicators do not necessarily predict runtime performance, because operator mix, parallelism, memory access and implementation determine measured speed [188, 189]. We therefore distinguish them from system-level measurements (wall-clock latency, throughput, peak memory), whose interpretation depends on batch size, sequence length, precision, device and software stack [189]. Offline methods report end-to-end or stage-specific latency and memory; streaming systems additionally report processing rate or response latency together with bounded memory as the stream grows [8, 149]. Energy is a relevant system metric [190], but none of the surveyed methods reports it, so we do not compare it.

Models are also rarely evaluated under identical input and modality conditions: two systems may claim the same GFLOPs per video while processing different frame counts, resolutions and modalities. EgoSchema’s intrinsic temporal length quantifies how much of a video must be processed to answer a question [34], and frame-sampling and streaming methods report accuracy against frame or time budgets [191], but these budgets are not standardized across papers.

We therefore use only values explicitly reported by each paper, record the corresponding model variant and input setting, and neither infer FLOPs or latency from architecture alone nor convert token-retention budgets into FLOPs; cross-paper comparisons serve only as indicative evidence. The quantitative comparisons emphasize 7B language backbones and approximately-8B configurations when the source reports a targeted efficiency mechanism. Parameter columns follow the source’s accounting boundary, which can include the full model or only the language backbone. The taxonomy also includes transferable mechanisms evaluated on larger hosts and standalone encoder methods; these do not enter a shared-host comparison unless their evaluation setting matches it.

Following the pipeline perspective of prior efficiency surveys [25, 26], we classify each reduction by its position in the forward pass:

1. Input construction and selection: selecting frames, patches, resolution or layouts before the encoder;

2. Encoder computation: changing feature-extraction operators, reducing intermediate tokens, or reusing and substituting encoder computation;

3. Encoded representations and connector: reducing encoder outputs or connector representations before the LLM;

4. LLM execution and state: reducing token processing or attention within the LLM, constructing summary tokens with its layers, or managing its KV cache.

We assign each mechanism to the stage whose computation it removes. Pooling applied after an encoder’s final block therefore counts as stage 3 even when it is implemented in the encoder, and whole-frame selection can occur after encoding, as in Frame-Voyager [140]. A method that reduces cost at several stages appears at each of them in Figure 4, and its reported end-to-end gain is not attributed to a single stage.

IV-B Input Construction and Selection

Frame sampling reduces the number of processed frames $T$ before the encoder runs, directly lowering encoder-side cost and the number of visual tokens ${\tilde{N}}_v$ injected into the language model. In encoder–connector–LLM pipelines, this upstream decision impacts both the cost of feature extraction and the LLM prefilling cost through the total context length $L=N_t+{\tilde{N}}_v\left(+{\tilde{N}}_a\right)$ (Section III-B); frames discarded at this stage cannot be recovered downstream. The two input families control temporal coverage and the spatial input budget, respectively. Temporal selectors may be query-free, using only the video signal, or query-aware, conditioning on the question or instruction. A proxy encoder or selector may process candidates that the target model never sees; its cost remains part of the selection pipeline.

IV-B1 Temporal Sampling and Selection

Fixed coverage sampling

Uniform or strided sampling remains the simplest query-free baseline: it is deterministic, model-free, and often strong. Recent controlled evaluation [192] confirms that frame-sampling choices alone can change video-QA results, and that uniform sampling can be the strongest strategy on Video-MME for some small VLMs [38]. Temporal Segment Networks (TSN) [98] introduced a stronger fixed-budget pattern by splitting the video into $K$ segments and sampling one snippet per segment for constant-cost temporal coverage. This “coverage under fixed $K$” idea remains a useful reference in later recognition and video-language pipelines [76, 193].

Content-based coverage

For minute-to-hour videos, temporal redundancy makes fixed windows particularly inefficient. Kernel Temporal Segmentation (KTS) [194] partitions a sequence of frame descriptors into segments. Their KVS summarizer adds trained category-specific SVM scoring to select summary segments. Later work [85] uses KTS to allocate samples before a downstream backbone for long-form classification and temporal localization. MaxInfo [89] uses proxy frame embeddings to maximize the geometric volume spanned by the selected subset. MGSampler [90] uses motion saliency and motion-uniform temporal coverage without a learned sampling policy.

Learned query-free selection

Learned query-free samplers use trained visual policies or scorers to adapt frame selection to each video without requiring a user query at inference time. AdaFrame [76] selects frames adaptively and performs early stopping using predicted future utilities. PEEK [91] distills caption-conditioned teacher rankings into a small visual temporal scorer, scoring frames from video embeddings alone. Earlier adaptive methods similarly learned to concentrate computation on informative video regions [195, 196, 197, 198, 199].

Query-conditioned relevance and diversity

Query-aware methods condition selection on the question or instruction, usually by scoring frame–text alignment and then enforcing diversity or coverage. Adaptive Keyframe Sampling (AKS) [11] combines prompt–frame relevance with temporal coverage under a fixed token budget. Q-Frame [92] uses a text-image matching model such as CLIP [43] to score frames and also adapts per-frame resolution to process more frames within the same budget. FOCUS [81] formulates keyframe selection as pure exploration in a multi-armed bandit, identifying informative temporal regions while processing only a small fraction of candidate frames. AdaRD-Key [77] maximizes a relevance–diversity objective and falls back to diversity-only selection when the query alignment is weak. Several 2025–2026 methods extend this training-free line. BOLT [78] samples frames by inverse-transform sampling over CLIP frame–query similarity. F2C [80] scores frame–query relevance to select anchor frames and extends them into coherent clips. T* [96] recasts temporal search as object-guided spatial search over frame mosaics, and reports that 8 selected frames outperform 32 uniform frames. QCA [93] allocates the frame budget across segments by query relevance and content variation, EFS [79] partitions the video into events and anchors selection on the most query-relevant frame per event, and GIFT [83] scores each frame’s global irreplaceability under the query, matching 64-frame uniform accuracy with 32 frames. KTV [86] combines query-free keyframe clustering at this stage with post-encoder token pruning (Section IV-D). LDDR [87] relaxes the binary keep/drop decision itself: it linearizes determinantal-point-process selection (from quadratic to linear complexity in the frame count) while jointly allocating per-frame resolution under an explicit token budget, applicable even to closed-source hosts. Related training-free methods explore scalable text-video similarity [200], sequential relevance-diversity allocation [201], semantic query decomposition [202], and lightweight moment retrieval for long-form VideoQA [203].

Learned and generative selectors

Other query-aware methods train explicit selectors. GenS [82] uses a separate VideoLLM to generate question-relevant frame selections for minute-to-hour videos, while HFS [84] optimizes a differentiable set-level objective combining relevance, coverage, and redundancy through Gumbel-Softmax [204] and student–teacher mutual learning. Qin et al. [94] train a 0.4B plug-in selector with reinforcement learning that transfers across seven LLM hosts and selects 8 of 128 candidate frames at less than half the selection latency of AKS [11]. Several recent selectors use reinforcement learning. TSPO [99] trains a temporal sampling policy with only 3.5M trainable parameters over frozen CLIP features using a GRPO-style (Group Relative Policy Optimization) objective and transfers it across hosts; ReFoCUS [95] optimizes a 1.3B policy with a logit-gap reward from the answering model, at a reported selection cost of 428 TFLOPs, 9 s and 5.3 GB over 512-frame inputs that its accuracy gains must amortize; and ViaRL [100] trains a 3B selector through iterated amplification to pick 8 of 128 candidate frames, though it reports no selector-overhead measurements. VideoITG [101] trains an 8B instructed temporal-grounding selector by supervised fine-tuning on automatically annotated data; its 32 selected frames match 64-frame uniform sampling while scanning 512 candidates and adding only 0.61 s, but the selector’s own size dominates any parameter-based efficiency accounting. Related learned methods include M-LLM-based frame selection [205], which trains a lightweight selector from pseudo-labels; K-frames [206], which predicts query-relevant coherent clips under arbitrary frame budgets; and FrameOracle [207], which predicts both which frames to retain and how many are needed.

IV-B2 Patch, Resolution, and Input-Layout Budgeting

Temporal selection leaves another choice: how much spatial detail to encode in each retained frame. Q-Frame [92] and LDDR [87] allocate per-frame resolution by relevance, while F2C [80] trades spatial resolution for longer clips under a fixed token budget. TS-LLaVA [97] combines several downsampled frames into a thumbnail grid before encoding, then samples additional encoded tokens in stage 3 (Section IV-D).

AutoGaze [102] selects multi-scale patches before the ViT using a 3M-parameter autoregressive selector. MeToM’s residual-guided patch merging [103] uses codec residual energy to identify connected low-information regions and average their patch embeddings before the heavy encoder blocks. Both reduce the input sequence those blocks process. MeToM additionally merges tokens after projection and inside the LLM; its reported end-to-end gain belongs to the combined pipeline. The idea has been explored in VATT [104], which randomly discards a fraction of the input patches and audio tokens before the transformer, and its ablation shows encoder GFLOPs falling with the drop rate at a growing accuracy cost on recognition benchmarks. It is a mechanism from before VideoLLMs, but it established that a video transformer tolerates a sparse input, which is the premise of the learned patch selection above.

IV-B3 Discussion and Synthesis

Frame sampling should be evaluated as a performance–budget trade-off, not as a single accuracy number. Clean comparisons fix the downstream model, frame budget, benchmark and split; otherwise the sampler, vision representation, connector and LLM capacity are confounded, an issue KFS-Bench [208] makes explicit by scoring coverage of the disjoint evidence scenes required for long-video QA alongside answer accuracy. Table II therefore reports only methods sharing a LLaVA-Video-7B [53], 64-frame protocol; selectors evaluated under other protocols (such as PEEK’s captioning setting [91]) are discussed in the text and excluded from the comparison. Query-free selections can be reused across questions. Query-aware methods gain up to 5 points on LongVideoBench [39] over uniform sampling when the question identifies sparse evidence, and TSPO [99] leads the table with only a 3.5M-parameter selector. Net savings still depend on whether avoided downstream work exceeds scorer cost, and the advantage largely disappears on Video-MME. Selectors evaluated at reduced budgets (VideoITG [101], GIFT [83]) match the 64-frame uniform reference with 32 selected frames.

媒体内容 · 前往原文查看

Method	Query-aware	Train-free	Frames	LongVideoBench	V-MME
Uniform baseline	–	–	64	58.9	64.4
MaxInfo [89]	no	yes	64	61.5	64.2
AKS [11]	yes	yes	64	62.7	65.3
AdaRD-Key [77]	yes	yes	64	62.9	–
FOCUS [81]	yes	yes	32–64†	63.5	65.4
EFS [79]	yes	yes	64	62.1‡	65.6‡
QCA [93]	yes	yes	64	62.9	66.1
TSPO [99]	yes	no	64	63.9	65.5

IV-C Encoder Computation

Encoder efficiency targets feature extraction before connector or LLM processing. We distinguish efficient architectures and operators, intermediate-token reduction, and computation reuse or substitution. Many architectural antecedents were evaluated on recognition or retrieval; their costs and accuracies must be kept separate from integrated VideoLLM results.

IV-C1 Efficient Encoder Architectures and Operators

Spatiotemporal backbones

Lightweight convolutional networks and hierarchical pooling-attention transformers reduce the cost of extracting video features. Among the convolutional backbones, TSM [118] inserts a parameter- and FLOP-free channel shift into a 2D CNN to capture temporal structure at roughly the cost of a 2D network; X3D [125] progressively expands a small 2D image architecture along its temporal, spatial, channel-width, and depth dimensions, selecting efficient configurations under increasing compute budgets; and MoViNet [113] pairs neural architecture search (NAS) with a stream-buffer that decouples memory from clip length for constant-memory streaming inference. The transformer-based group reduces token resolution inside the encoder: MViT [114] and MViTv2 [115] use multi-head pooling attention to progressively pool spatiotemporal tokens while widening channels, MViTv2 adding decomposed relative position and residual pooling for 82.9 vs. 82.7 on Kinetics-400 at 51M vs. 88M parameters and a third of the inference compute of Video Swin (Table III); Hiera [106] strips MViTv2 of its specialized components and leans on strong masked auto-encoder pretraining, yielding a simpler backbone about $2\times$ faster on video (40.8 vs. 20.5 clips/s) at 5.0 points higher video accuracy than MViTv2-L; Video Swin [121] restricts self-attention to shifted local 3D windows; and UniFormer [119] couples convolution-like local aggregation in shallow layers with global attention in deeper layers, with UniFormerV2 [120] equipping a frozen pretrained image ViT with lightweight video-specific UniBlocks.

State-space operators

State-space encoders replace quadratic self-attention with selective state-space models for linear-time video encoding. VideoMamba [122] uses bidirectional Mamba blocks [209] to process spatiotemporal tokens, reporting $6\times$ higher throughput and $40\times$ lower GPU memory than TimeSformer-Ti at 64 frames (A100-80G, batch size 128). VideoMamba-ST [123] adapts the scan order to video structure (we use the -ST suffix because Park et al. also name their model VideoMamba). VideoMambaPro [124] addresses information leakage in the backward scan through masked backward computation and residual connections, improving Kinetics-400 accuracy by 1.6 points over VideoMamba-M at matched $32\times {224}^2$ input (84.0 vs. 82.4) with slightly fewer parameters and FLOPs.

Compact and sparse encoders

Compact encoders reduce the cost of feature extraction, either by distilling CLIP-style encoders [43] or by designing the encoder to emit fewer tokens. On the distillation side, TinyCLIP [117] combines affinity-mimicking distillation with weight inheritance to shrink CLIP encoders, MobileCLIP [109] uses multi-modal reinforced training to produce fast image–text encoders (MobileCLIP2 [110] strengthens the teacher ensembles at 1.5–20 ms on-device encoder latencies), and MobileViCLIP [111] carries this to video with a compact mobile video–text encoder: MobileViCLIP-Small runs $55.4\times$ faster than InternVideo2-L14 on mobile hardware at similar zero-shot retrieval performance. On the token-budget side, FastVLM [105] introduces a hybrid convolution–transformer encoder that downsamples aggressively to emit far fewer high-resolution visual tokens, reporting $85\times$ faster time-to-first-token with a $3.4\times$ smaller vision encoder than LLaVA-OneVision-0.5B [210] at ${1152}^2$ input, while LiteFrame [107] distills a compact VideoLLM vision encoder that emits compressed tokens and cuts end-to-end latency by $35\%$ relative to InternVL3-8B while processing $8\times$ more frames. MoE-ViE [112] scales the encoder sparsely, activating 1.1B of 3.5B parameters per token through a fine-grained mixture of experts to match a dense encoder $1.7\times$ its size at roughly three quarters of its latency, with video-benchmark evidence on an 8B host.

Oryx [116] also changes the encoder: native-resolution processing avoids fixed-resolution tiling, followed by a dynamic compressor in stage 3 (Section IV-D). Finally MMV [108] adopts TSM for inexpensive temporal modeling.

IV-C2 Encoder-Internal Token Reduction

ToMe [128] merges similar tokens through bipartite matching between encoder blocks, raising ViT-L video throughput by $2.2\times$ for a $0.2$%–$0.3$% accuracy drop. EarlyTom [126] merges frame features between encoder blocks, then selects spatial tokens after encoding. Its encoder reduction and post-encoder selection therefore occupy stages 2 and 3 (Section IV-C and IV-D). ResidualViT [127] combines intermediate-token reduction with temporal reuse, discussed below. Because these methods act between blocks, the early blocks still process the full sequence; the patch selection of Section IV-B instead removes tokens before the first block, so the encoder never sees them.

IV-C3 Encoder Computation Reuse and Substitution

ResidualViT [127] propagates a residual subset of tokens across frames for temporally dense encoding, reducing per-frame encoding cost by $53$%–$56$% within 1.7 points of CLIP R@1 on Charades-STA. STC [130] combines STC-Cacher, which reuses cached ViT features for temporally similar content, with STC-Pruner, which compresses the encoded sequence before LLM input. With cached features reused for $75\%$ of tokens, the combined method reduces ViT-encoding latency by $24.5\%$ and LLM prefilling latency by $45.3\%$. MoViNet’s stream buffer [113] and the online variant of TSM [118] retain temporal features between successive inputs.

CoPE-VideoLM [129] runs the image encoder only on the few frames a video codec stores in full, and encodes the remaining frames from the motion and residual information the codec already provides, using a delta-encoder of under 15M parameters in place of dense RGB encoding.

媒体内容 · 前往原文查看

Method	Year	Params (B)	GFLOPs$\times$v	K-400	K-600	MiT	AS	UCF	HMDB
TSM [118]	2019	0.024	65$\times$1	74.7	—	—	—	95.9	73.5
MMV [108]	2020	0.094	—	—	70.5	—	30.9	95.2	75.0
X3D [125]	2020	0.011	35.84$\times$10	78.4	81.9	—	—	—	—
MoViNet [113]	2021	0.031	386$\times$1	—	84.8	39.9	—	—	—
MViT [114]	2021	0.037	170$\times$5	80.2	83.4	—	—	—	—
VATT [104]	2021	0.155	15 020t	79.9	80.8	37.8	39.3	—	—
MViTv2 [115]	2022	0.051	225$\times$5	82.9	85.5	—	—	—	—
Video Swin[121]	2022	0.088	282$\times$12	82.7	—	—	—	—	—
UniFormer [119]	2022	0.050	3108t	83.0	84.9	—	—	—	—
Hiera [106]	2023	0.213	413$\times$15	87.3	—	—	—	—	—
UniFormerV2 [120]	2023	0.354	75 300t	90.0	90.1	47.8	—	—	—
VideoMamba [122]	2024	0.074	403$\times$12	82.4	—	—	—	88.2	60.8
VideoMamba-ST [123]	2024	0.027	68$\times$15	77.7	—	—	—	—	75.7
VideoMambaPro [124]	2025	0.072	4700t	84.0	—	—	—	91.6	63.2

媒体内容 · 前往原文查看

IV-C4 Discussion and Synthesis

Convolutional backbones occupy the low-compute regime, while pooling-attention transformers span the widest accuracy range and reach the highest absolute accuracies. State-space backbones are competitive (VideoMambaPro [124] reaches 84.0 top-1 at 4.7 TFLOPs, above the global-attention baselines at comparable cost) but none matches Hiera [106] or UniFormerV2 [120] at any compute, as shown in Figure 5. The evaluation protocol also reshapes the apparent trade-off: MViTv2 [115] and Video Swin [121] report nearly identical Kinetics-400 accuracy, yet 1.13 versus 3.38 TFLOPs because they evaluate with five versus twelve views. Table III therefore supports comparisons between reported operating points, not attribution of gaps to architecture alone; and encoder-only FLOPs do not establish end-to-end VideoLLM efficiency, which also depends on the output token count and the downstream connector and LLM.

Several recent encoder mechanisms, including EarlyTom, STC and CoPE, are evaluated in VideoLLMs and report time-to-first-token or end-to-end latency. Compact image encoders and standalone retrieval methods were evaluated on different tasks, so their gains cannot be transferred numerically to VideoLLM QA.

IV-D Encoded Representations and Connector

This stage reduces encoded representations before the answering LLM consumes them. The reduction may act on whole frames, individual tokens, pooled grids, latent representations or a maintained memory bank. Its immediate benefit is a smaller LLM input; encoding costs have already been paid unless a separate upstream mechanism also reduces them.

IV-D1 Selection and Merging of Encoded Representations

This family removes or fuses encoded tokens. Many methods operate on a frozen host, while others train the system around the reduction. VisionZip [153] keeps only the most informative tokens (retaining $6.6\%$ of them at a $7.8\times$ prefilling speed-up), LLaVA-PruMerge [144] adaptively prunes and merges for $14\times$ average visual-token compression, and Chat-UniVi [131] uses parameter-free clustering to merge tokens. PruneVid [12] and HoliTom [142] exploit spatiotemporal redundancy (HoliTom runs at roughly $10\%$ of the baseline FLOPs), and LongVU [88] combines frame selection before SigLIP [215] with query-conditioned pooling and token pruning after encoding. FlashVID [137] combines attention- and diversity-based token selection with tree-based spatiotemporal merging, holding $99.1\%$ relative accuracy at $10\%$ retention with a $6.3\times$ prefilling speed-up, and EchoPrune [13] drops tokens that are reconstructible from previous frames, interpreting them as temporal echoes, which allows using up to $20\times$ more frames under a fixed token budget. DyToK [135] introduces a budget-allocation policy: an assistant model supplies a query-conditioned per-frame prior, which is converted into per-frame retention ratios. Where the reduction happens depends on the compressor it drives: before the LLM with VisionZip [153], inside it with FastV [173], and at both with DyCoke [133]. The assistant model’s forward pass adds cost that a complete comparison must count. Recent papers differ mainly in how the token budget is allocated across time. FastVID [136] partitions the video into temporally ordered segments and prunes by density within each, reducing FLOPs to $8.3\%$ for a $7.1\times$ prefilling speed-up at $98\%$ retained accuracy; LLaVA-Scissor [145] compresses through semantic connected components; and VidCom2 [151] adapts per-frame compression intensity to frame uniqueness, reducing LLM-generation latency by $70.8\%$ at a quarter of the tokens. Segment-level budget allocation recurs in MMG-Vid [146] (marginal-gain maximization, $3.9\times$ prefilling speed-up at $25\%$ retention), OTT-Vid [147] (optimal-transport cost between neighboring frames), InfoMerge [143] (second-order temporal fingerprints with spectral-entropy budgets, $4.2\times$ prefilling speed-up at $15\%$ tokens), DynaTok [134] (an EMA novelty memory with positional-bias-aware spatial selection), and ForestPrune [139] (globally optimized pruning over spatio-temporal token forests). MeToM [103] allocates post-projector token budgets from groups of pictures packet sizes, then merges redundant tokens across time and within frames. This stage complements its input-patch merging and LLM-layer merging; the reported $2.65\times$ time-to-first-token speed-up measures their combined effect. Related methods condition on the query or train the reduction: KTV [86] runs video through an image-only VLM without training by clustering frames into keyframes and then pruning each keyframe’s tokens by importance and redundancy, LGTTP [216] prunes tokens outside a query-predicted temporal window through a trained auxiliary classifier, and DynTok [217] trains the grouping-and-merging step into the model itself to avoid a training–inference mismatch.

Frame-Voyager [140] selects whole frames at this stage. All candidate frames first pass through the host visual encoder and projector; pooled features and the query then enter a scorer built from frozen bottom LLM layers and trained reward heads. The selected frames supply the answering context. This reduces the context relative to answering over all candidates, but does not spare their initial encoding. Its Appendix C reports $27.6\%$ higher latency than uniform sampling in the tested setting, illustrating why selection quality and net speedup require separate comparisons.

FlexSelect [138] also selects tokens before a final answering pass. Its base variant scores encoded frame sets through partial host-model forwards, then aggregates the selected tokens; FlexSelect-Lite replaces that scorer with a trained lightweight selector.

VideoChat-Flash [152] merges similar clip tokens before the LLM and progressively drops tokens inside its layers. TimeChat-Online [150] drops encoded tokens whose content is unchanged between successive frames ($82.8\%$ reduction at approximately $98\%$ retained streaming accuracy and $1.76\times$ faster responses); the dropping rule also transfers without training to Qwen2.5-VL. StreamingTOM [149] combines causal temporal selection and merging here with quantized memory and retrieval in stage 4 (Section IV-E).

Audio can supply the selection signal. OmniZip [15] and DASH [132] use audio to guide visual-token reduction; OmniZip also compresses the audio stream. Its Qwen2.5-Omni-7B configuration reports a $3.42\times$ speed-up and $1.4\times$ memory reduction at $35\%$ token retention (Table IV). The audio encoder that supplies the guidance is part of the cost and should appear in the reported savings.

IV-D2 Grid Pooling and Downsampling

Spatial or temporal downsampling reduces the encoded sequence through a prescribed grid structure: pixel-shuffle in InternVL2.5 [59], adaptive pooling in PLLaVA [157], a SlowFast two-stream projector in SF-LLaVA [160], the spatial-temporal convolution connector of VideoLLaMA 2 [71], and a learned Mamba temporal connector in STORM [148] that cuts computation by up to $8\times$ and decoding latency by $2.4$–$2.9\times$ at a fixed frame count. Several efficiency-first VideoLLM architectures make this stage their central design. NVILA [156] scales spatial and temporal resolution first, then compresses tokens, for $1.6$–$2.2\times$ lower prefilling and $1.2$–$2.8\times$ lower decoding latency than comparable open VLMs; PVC [158] uses temporal attention to enrich frame features before compressing each frame to 64 tokens; TS-LLaVA [97] builds a fixed 3,456-token budget from a detail thumbnail plus tokens sampled across 50 frames.

VideoScan [162] pools each frame into a semantic-carrier token before the LLM and learns a KV propagation policy inside it. Qwen2-Audio [159] is an audio-only precedent: stride-2 pooling follows the audio encoder’s transformer blocks. Baichuan-Omni [154] uses convolutional downsampling, while HyperCLOVA X 8B [155] adopts MambaMia to reduce the audio rate from 25 Hz to 1 Hz after its adapter. The latter report does not describe how its visual tokens are reduced, so we classify only the audio downsampling; its reported visual-token budget and training-cost savings are not attributable to a described mechanism.

IV-D3 Latent Resampling and Compact Representation Construction

Learned resamplers construct a compact representation from the encoder outputs, often through cross-attention with a small set of latent queries. The Perceiver Resampler of Flamingo [165] and the Q-Former of Video-LLaMA [42] are early influential examples of learned resampling; LLaMA-VID [55] combines a query-conditioned context token with pooled content, reaching two tokens per frame in its compressed setting, and LLaVA-Mini [166] reaches a single vision token via modality pre-fusion ($-77\%$ FLOPs). BLIP-3-Video [163] abstracts an entire video into 16–32 learned tokens; Quicksviewer [167] learns nonuniform temporal “cubes” through Gumbel-Softmax and resamples 64 tokens per cube for a $45\times$ overall compression; and VidCompress [161] pairs a memory-enhanced compressor emitting one token per frame with a text-perceived Q-Former branch. VQToken [169] replaces the continuous bottleneck with a discrete one: adaptive vector quantization maps ViT embeddings onto a learned codebook, with a token hash preserving spatiotemporal position, shrinking the video stream to $0.07\%$ of its tokens at a $0.66$-point drop on NExT-QA.

Oryx [116] combines native-resolution encoding with an on-demand cross-attention compressor at $1\times$–$16\times$ reduction. Audiovisual resamplers follow the same principle: FAVOR [164] uses a windowed causal Q-Former to enforce a joint budget, and video-SALMONN [168] queries features at fine (approximately 0.5 s) and coarse (approximately 5 s) temporal resolutions. Table IV reports their 7B configurations.

IV-D4 Representation-Memory Compression and Retrieval

This family compresses a maintained history of encoded representations outside the answering LLM. MovieChat [62] merges features as its short-term buffer fills, MA-LMM [63] maintains compressed visual and query memory banks around the Q-Former, and $\infty$-Video [67] uses a continuous-time long-term representation with resampling. VidCompress [161] and the Token Turing Machine variant of BLIP-3-Video [163] also maintain state while constructing compressed outputs.

Flash-VStream [41] maintains a two-part memory combining compact context with selected high-resolution details, while VideoLLaMB [171] propagates recurrent memory bridges across semantic segments. AdaCM² [170] prunes the Q-Former video cache using cross-modal attention, processing videos beyond two hours with a reported $65\%$ reduction in GPU memory. This cache belongs to the connector; it is distinct from the answering LLM’s KV cache.

IV-D5 Discussion and Synthesis

Table IV summarizes these methods. Because the quoted accuracies come from different language backbones and host VLMs, we use its rows only as indicative evidence. Where the literature provides a controlled comparison, we report it separately: Table V compares the training-free methods that HoliTom [142] re-ran on a single frozen LLaVA-OneVision-7B host [210] at matched token budgets, a comparison of post-encoder and joint-stage methods alongside Tables II and VII. At a $25\%$ budget the compared methods with post-encoder reduction stay within $1.5\%$ of the uncompressed baseline average while DyCoke [133], which combines pre-LLM temporal merging and decoder KV management (Section IV-E), loses over $7\%$. At $10\%$ the near-tie breaks down, and methods that model temporal redundancy explicitly (PruneVid [12], HoliTom [142]) degrade far more gracefully than spatial-only selection (VisionZip [153]). The lower block adds two methods whose own runs reproduce the same host and harness: FlashVID [137] matches HoliTom’s near-lossless behavior at both budgets, while EarlyTom [126], which prunes inside the vision encoder and therefore also cuts encoding compute, stays competitive at $25\%$ but sits between spatial-only and temporal-aware methods at $10\%$. Connector-stage reduction lowers LLM prefilling cost without reducing the cost of encoding the retained frames; the saved budget can instead widen temporal coverage, as in EchoPrune [13], so fixed-input and fixed-downstream-budget evaluations measure different benefits.

媒体内容 · 前往原文查看

Method	Year	Params (B)	LLM/host	Retained budget	NX-QA	MSR	MSVD	MVB	VME	ES	ActNet
3a. Selection and merging of encoded representations
VisionZip [153]	2024	7h	Video-LLaVA	6.6%	—	52.1	63.5	—	—	—	43.0
LLaVA-PruMerge [144]	2024	7h	Video-LLaVA	256/img	—	59.3	71.1	—	—	—	47.7
Chat-UniVi [131]	2024	7	Vicuna-1.5	44%	—	55.0	69.3	—	—	—	46.1
LongVU [88]	2024	7	Qwen2	45%	—	—	—	66.9	60.6	67.6	—
PruneVid [12]	2025	7h	LLaVA-OV	15–17%	—	—	—	57.5	58.6	59.5	—
HoliTom [142]	2025	7	LLaVA-OV	10%	—	—	—	57.3	56.8	61.2	—
FlashVID [137]	2026	7h	LLaVA-OV	10%	—	—	—	57.4	57.8	60.0	—
EchoPrune [13]	2026	7h	LLaVA-OV	10%/320f	—	—	—	—	61.8	60.4	—
FastVID [136]	2025	7h	LLaVA-OV	25%	—	—	—	56.3	58.0	—	—
LLaVA-Scissor [145]	2025	7h	LLaVA-OV	10%	80.0	—	—	57.9	55.2	57.5	47.8
VidCom2 [151]	2025	7h	LLaVA-OV	25%	—	—	—	57.2	58.6	59.7	—
MMG-Vid [146]	2025	7h	LLaVA-OV	25%	—	—	—	56.7	58.6	—	—
TS-LLaVA [97]	2024	7	Vicuna-1.5	3456/50f	66.5	65.1	79.0	45.5	—	50.2	56.7
VideoChat-Flash [152]	2025	7	Qwen2	16/frame	—	—	—	74.0	65.3	—	—
StreamingTOM [149]	2025	7	LLaVA-OV	25.5%	—	—	—	—	59.9	63.7	—
TimeChat-Online [150]	2025	7	Qwen2.5-VL	$\sim$17%	—	—	—	—	62.5	—	—
OmniZip [15]	2025	7h	Qwen2.5-Omni	35%	—	—	—	—	66.1	—	—
DASH [132]	2026	7	Qwen2.5-Omni	25%	—	—	—	—	66.0	—	—
3b. Grid pooling and downsampling
VideoLLaMA 2 [71]	2024	7h	Mistral	50%	—	—	70.9	54.6	47.9	51.7	50.2
InternVL2.5 [59]	2024	8.1	InternLM2.5	25%	—	—	—	72.0	64.2	—	—
PLLaVA [157]	2024	7	LLaVA-NeXT	25%	—	62.0	76.6	—	—	—	56.3
SF-LLaVA [160]	2024	7	LLaVA-NeXT	3680 total	64.2	65.8	79.1	—	—	47.2	55.5
STORM [148]	2025	7	Qwen2	25%	—	—	—	71.3	63.4	—	—
NVILA [156]	2024	8	Qwen2	1/8	82.2	—	—	68.1	64.2	—	60.9
PVC [158]	2024	8	InternLM2.5	64/frame	82.0	—	—	73.8	64.1	59.6	57.1
VideoScan [162]	2025	7	LLaVA-Video	1/frame	—	—	—	48.9	53.7	—	—
Baichuan-Omni [154]	2024	7	own	182–546/video	—	—	72.2	60.9	58.2	58.8	58.6
HyperCLOVA X 8B [155]	2026	8	own	1/s audio	—	—	—	—	58.2	—	—
3c. Latent resampling and compact representation construction
LLaMA-VID [55]	2023	7	Vicuna	2/frame	—	57.7	69.7	—	—	—	47.4
LLaVA-Mini [166]	2025	7	Vicuna-1.5	1/frame	—	59.5	70.9	44.5	—	51.2	53.5
BLIP-3-Video [163]	2024	4	Phi-3-Mini	32/video	76.4	60.0	77.7	54.9	—	—	55.7
Quicksviewer [167]	2025	8	Qwen2.5-7B	64/cube	77.5	—	—	55.6	56.9	—	47.6
VidCompress [161]	2024	7	Vicuna	1/frame+QF	—	57.7	68.9	46.9	43.0	—	48.3
Oryx [116]	2024	7	Qwen2	1/4–1/16	81.9	—	—	63.9	58.3	—	—
FAVOR [164]	2023	7h	Vicuna	160/25 s	42.5	—	—	—	—	—	—
video-SALMONN [168]	2024	7h	Vicuna-1.5	160/25 s	42.5	—	—	—	—	—	—
3d. Representation-memory compression and retrieval
MovieChat [62]	2024	7h	Vicuna	576 mem	—	52.7	75.2	—	—	—	45.7
MA-LMM [63]	2024	7	Vicuna	32 mem	—	48.5	60.6	—	—	—	49.8
$\infty$-Video [67]	2025	7	V-LLaMA/VC2	—	41.1	—	—	—	42.4	46.8	—
Flash-VStream [41]	2025	7	Qwen2	11.5K/stream	—	—	—	65.4	61.2	68.2	—
VideoLLaMB [171]	2025	7	Vicuna-1.5	32 mem/seg	71.1	—	—	52.5	41.4	53.8	—
4a. Decoder token pruning and merging
STTM [177]	2025	7h	LLaVA-OV	50%	80.4	—	—	—	60.7	61.7	—
4c. LLM-computed summary tokens
VoCo-LLaMA [187]	2024	7	Vicuna	2/frame	—	61.1	72.3	—	—	—	47.9
Video-XL [186]	2024	7	Qwen2	KV 1/16	—	—	—	55.3	55.5	—	—

媒体内容 · 前往原文查看

Method	Tokens kept	FLOPs	MVBench	EgoSch.	LongVideoBench	V-MME w/o	Avg. %
LLaVA-OV-7B (base)	100%	100%	58.3	60.4	56.4	58.6	100
DyCoke [133]	25%	21.3%	53.1	59.5	49.5	54.3	92.6
VisionZip [153]	25%	21.3%	57.9	60.3	56.5	58.2	99.7
PruneVid [12]	25%	21.3%	57.4	59.9	55.7	57.4	98.6
FastVIDm [136]	25%	21.3%	56.5	—	56.3	58.0	—
HoliTom [142]	25%	17.4%	58.4	61.2	56.7	58.9	100.7
VisionZip [153]	10%	8.3%	53.5	58.0	49.3	53.4	91.6
PruneVid [12]	10%	8.3%	56.2	59.8	54.5	56.0	96.9
FastVIDm [136]	10%	8.3%	55.9	—	56.3	57.3	—
HoliTom [142]	10%	6.9%	57.3	61.2	56.3	56.8	99.1
FlashVID [137]	25%	—	58.0	60.4	56.8	59.2	100.3
EarlyTom [126]	25%	44.2%e	57.4	60.5	56.3	58.5	99.7
FlashVID [137]	10%	—	57.4	60.0	56.5	57.8	99.1
EarlyTom [126]	10%	39.0%e	56.5	60.1	52.4	55.8	96.2
VidCom2m [151]	25%	—	57.2	59.7	54.9	58.6	99.6p
MMG-Vidm [146]	25%	—	56.7	—	56.6	58.6	99.5p

IV-E LLM Execution and State

The final pipeline stage targets already-projected visual tokens inside the language model, where they dominate the context length $L$ that drives prefilling cost and KV-cache memory (Section III-B). We distinguish token pruning and merging, sparse attention, learned summary tokens, KV compaction, and KV offloading or retrieval.

IV-E1 Decoder Token Pruning and Merging

The first family reduces how many visual tokens propagate through the decoder layers. FastV [173] shows that visual tokens receive little attention in deeper decoder layers and exploits this finding by pruning the lowest-attention half after an early layer, roughly halving prefilling FLOPs. HieraVid’s controlled video re-run at $39.3\%$ FLOPs costs 3–5 points across MVBench, NExT-QA, EgoSchema and Video-MME (Table VII). SparseVLM [176] performs progressive, text-guided pruning across decoder layers, retaining fewer than $10\%$ of the visual tokens, while a recycling step compresses selected pruned tokens into a smaller set of representative tokens. FrameFusion [174] and HieraVid [141] specialize the idea for video by first merging temporally redundant tokens across frames and only then pruning by importance: FrameFusion as a two-phase merge-then-prune cascade reporting $1.6$–$3.6\times$ end-to-end speed-ups, and HieraVid as a three-level segment/frame/layer hierarchy that cuts prefilling FLOPs to roughly a quarter of baseline at $30\%$ token retention while retaining about $98\%$ of average accuracy. As decoder backbones themselves diversify, Jiang et al. [175] extend the family to Mamba–Transformer hybrids [218]: they show that recurrent state layers compress the information carried by removed tokens into their hidden state, and their progressive, query-conditioned schedule yields a $3.8$–$4.2\times$ prefilling speed-up at a $25\%$ token budget at near-baseline accuracy, improving with light finetuning. In the same hybrid direction, TimeViper [178] folds visual-token information into the instruction tokens at two decoder depths and drops the visual tokens, reaching over 10,000 frames with a $15.7$% shorter prefill at 4,096 frames for a 1–2 point accuracy cost. STTM [177] merges quadtree-derived spatial tokens across time at an early LLM layer. AdaTP [172] corrects attention-sink and positional biases in pruning scores, retaining baseline accuracy at $27\%$ of FLOPs. PruneVid [12], HoliTom [142], FlashVID [137], VideoChat-Flash [152], HieraVid [141], and MeToM [103] combine reduction before the LLM with reduction inside its layers.

IV-E2 Sparse Decoder Attention

MMInference [184] accelerates prefilling by skipping attention pairs while retaining the token sequence. A modality-aware permutation gathers the grid-structured sparse attention induced by video into GPU-friendly blocks, yielding up to $8.3\times$ prefilling speed-up at million-token contexts with at most 0.4-point accuracy differences across the reported 7B hosts. ReKV’s sliding-window attention [185] also restricts the attended context during stream encoding, combined with cache offloading and retrieval below.

IV-E3 LLM-Computed Summary Tokens

VoCo-LLaMA [187] learns compression tokens whose representations are computed by the LLM’s own layers under an attention constraint. Subsequent processing uses these compact summaries in place of the full visual context. This differs from a Q-Former or Perceiver resampler operating before the language model. Video-XL [186] condenses each interval’s visual KV pairs into summarization tokens inside the LLM, reaching 2,048 frames on one A100, with successors pushing past 10,000 frames through reconstructive compression and task-aware KV sparsification [219, 220].

IV-E4 KV-Cache Compaction

KV compaction reduces stored state through eviction, merging or quantization. Eviction removes entries; quantization reduces the precision of those retained. DyCoke [133] dynamically evicts the least-attended visual tokens from the KV cache at each decode step, on top of a prefilling temporal-merging stage, for a $1.5\times$ inference speed-up and $1.4\times$ memory reduction against its baseline VideoLLM. VidKV [14] quantizes the visual KV cache to mixed precision ($\phantom{}\approx 1.5$-bit keys and $1.58$-bit values) and finds that, unlike text LLMs, the value cache of video models is better quantized per channel than per token, with almost no performance drop against FP16 on six benchmarks with LLaVA-OneVision-7B [210] and Qwen2.5-VL-7B [221]. ReTaKe [181] couples keyframe-level pruning (DPSelect) with pivot-guided KV eviction (PivotKV) for $8\times$ context compression, fitting 2,048 frames into a 16K context on Qwen2-VL-7B with a $20\%$ lower time-per-output-token, and AdaReTaKe [222] adapts the compression ratio across time and layers. MEDA [180] allocates per-layer KV budgets from cross-modal attention entropy, reaching $72\%$ KV-memory reduction and $2.82\times$ faster decoding on multimodal long-context suites. InfiniPot-V [179] evicts entries by temporal redundancy and value norms whenever its budget fills, reporting up to $94\%$ lower peak GPU memory. StreamMem [183] uses attention from generic proxy queries to compress the cache without the eventual user question, while VideoScan [162] learns which KV state to propagate alongside its pooled carrier inputs. Image-only MLLMs have parallel lines of work on visual-token withdrawal [223], layer-wise dropping [224] and KV eviction [225], which several of the video methods above adapt.

IV-E5 KV-Cache Offloading and Retrieval

Offloading preserves historical state outside GPU memory, then retrieves only the relevant portion for answering. ReKV [185] encodes the stream with sliding-window attention, offloads KV blocks to CPU RAM or disk, and retrieves query-relevant blocks at question time. StreamKV [182] combines per-segment cache compression with question-conditioned retrieval. StreamingTOM [149] stores quantized groups and selectively dequantizes relevant groups at generation time. These methods can bound active GPU state while allowing total stored history to grow. Table VI compares the reported streaming and offline long-video protocols.

媒体内容 · 前往原文查看

Offline long video, Qwen2-VL-7B
Method	KV budget	V-MME w/o	MLVU	LongVideoBench	EgoSch.
Full KV cache (range of reproductions)	100%	63.3–63.9	63.9–65.8	55.6–58.8	65.2
ReTaKe [181]	$8\times$ compr.	63.9	69.8	57.7	—
InfiniPot-V [179]	6K tokens	62.8	65.8	58.4	65.6
StreamMem [183]	6K tokens	62.1	65.9	—	67.2
Streaming QA, LLaVA-OneVision-7B (RVS-Ego / RVS-Movie), StreamMem reproduction
Method	Mechanism	RVS-Ego	RVS-Movie	Peak memory
Full KV / backbone baseline	—	56.2–60.1	43.0–53.4	37.5 GB
ReKV [185]	KV offload + retrieval	63.7	54.4	38 GBo
Flash-VStream [41]	learned fixed memory	57.0	53.1	—
InfiniPot-V [179]	capped KV eviction	57.9	51.4	27.8 GB
StreamMem [183]	query-agnostic KV memory	57.6	52.7	$<$28 GBc

^oGPU-resident peak with internal retrieval; ReKV additionally offloads 18.8 GB per stream-hour to CPU RAM or disk. ^cReported as the experiment’s memory constraint rather than a measured peak.

IV-E6 Discussion and Synthesis

Cross-method comparison at this stage is intrinsically limited: each method reports retained accuracy against its own backbone and baseline at a different token budget. Table VII gives the one controlled same-backbone comparison available, covering methods with decoder-layer reduction on LLaVA-Video-7B [53]: HieraVid [141] uses 24.5% of the baseline prefilling FLOPs while remaining within 0.2–2.1 points across the five reported settings, outperforming FastV [173] at a larger budget and FrameFusion [174] at a similar one, supporting temporal merging and pruning under this particular protocol. HieraVid also reduces tokens before the LLM, so this comparison does not isolate its decoder operation. Decoder-layer reduction primarily cuts prefilling computation, whereas visual KV-cache compression targets memory and latency during decoding, so the two mechanisms compose; KV eviction, merging and quantization have different effects on available evidence, and their published results do not share the protocol of Table VII. The streaming systems of Table VI expose a retrieval–eviction trade-off: ReKV’s [185] retrieval preserves streaming accuracy best but keeps peak GPU memory near the full-cache level, whereas hard-capped eviction (InfiniPot-V [179], StreamMem [183]) trades roughly six RVS-Ego [41] points for a constant memory ceiling about 10 GB lower. Sparse-attention prefilling (MMInference [184]) reduces a different part of the workload. Combining it with token or cache reduction requires checking compatibility and measuring the joint system; separate speedups cannot be multiplied.

媒体内容 · 前往原文查看

Method	FLOPs	MVBench	NExT-QA	EgoSch.	VME w/o	VME w/
LLaVA-Video (base)	100%	60.4	80.2	59.4	64.1	71.4
FastV [173]	39.3%	56.6	77.2	55.1	59.3	66.7
FrameFusion [174]	23.8%	56.7	78.8	56.8	61.9	70.1
HieraVid [141]	24.5%	58.3	79.9	59.2	62.3	70.8

V Discussion and Future Directions

Convergent trends across mechanisms. Several mechanism families report near-baseline accuracy at 25% visual-token retention: pixel-shuffle [59], pooling [157], temporal compression [148], audio-guided pruning [132], and decoder-layer hierarchies [141]. These results come from different hosts and protocols, so they do not imply that every VideoLLM can discard 75% of its visual tokens without loss; controlled results in Table V show that method choice is more important at 10% retention.

Two patterns emerge from placing methods by where they remove computation. First, selection is not always upstream: query-conditioned selectors such as Frame-Voyager [140] and FlexSelect [138] encode every candidate before choosing, so they shorten the LLM context but spare no encoder work, and Frame-Voyager reports higher latency than uniform sampling. Second, the strongest 2025–2026 results combine stages, reducing tokens before the LLM and again inside it [12, 142, 137, 141, 103]. Their end-to-end gains cannot be attributed to either stage, and only HoliTom [142] reports the ablation that separates them. A nominal reduction ratio therefore says little about where the savings occur or which component produced them.

Cross-stage bottleneck shifts are a shared observation in prior efficiency surveys: Zhang et al. [25] analyze interactions among encoding, prefilling and decoding, while Wu et al. [26] discuss global resource allocation across compression stages. The video-specific evidence reviewed here shows how this interaction affects the choice between encoding fewer frames, representing each frame more cheaply, and compressing the resulting context. Once downstream compression reduces the LLM token load, recent methods move upstream again by pruning inside the encoder [126], caching features across similar frames [130], or consuming codec primitives [129].

Some methods reinvest the saved compute: under a fixed LLM budget, token reduction can admit $10$–$20\times$ more frames [13, 137], allocate tokens continuously across frames [135], or support longer training contexts [175], so compression may improve accuracy by increasing temporal coverage. Evidence for audiovisual efficiency remains comparatively sparse: reported costs often omit the audio encoder and modality ablations are uncommon, making the benefit and cost of audio-guided selection hard to isolate.

The four stages are unevenly represented: reduction of encoded representations attracts the most papers, while the LLM-side families are small. The first connector-side and decoder-side reductions applied to video were image-only VLM methods evaluated frame by frame, such as FastV [173], SparseVLM [176], VisionZip [153], and LLaVA-PruMerge [144]. In contrast, the 2025–2026 methods in the same families exploit temporal redundancy directly, merging tokens across frames [174, 142, 151] or evicting cache entries by inter-frame similarity [133, 181]. The LLM-side families have not completed this move: KV eviction, quantization and sparse attention were developed for text-only LLMs and transfer to video with little modification, so fewer video-specific papers are needed to cover the same ground. Yet once pre-LLM compression has removed redundant tokens, the cost that remains is decoding memory and cache growth under multi-turn and streaming use, which only LLM-side mechanisms address, so we expect these families to grow fastest. Decoder backbones are also diversifying beyond dense transformers [175, 178, 218], making evidence preservation across attention and recurrent state a further evaluation question.

From action recognition to question answering. Read chronologically, the comparison tables of Section IV document a shift both in how efficiency is achieved and in how it is evidenced. Vision-encoder methods (Table III), proposed mostly between 2019 and 2023, evaluate on action recognition (Kinetics [27], Moments in Time [211], UCF101 [213]) and report inference GFLOPs per view under an explicit input protocol, whereas the later pipeline stages evaluate almost exclusively on video question answering (Tables II, IV and VII contain no classification benchmark), moving from GPT-assisted open-ended scoring in 2023–2024 to cheaper, less judge-dependent multiple choice in 2025–2026. The shift is in fact stage-dependent and not purely chronological: state-space encoder papers from 2025 still evaluate on Kinetics [124]. Because each stage evaluates on different tasks and metrics, efficiency progress cannot be compared consistently across years, and an efficient Kinetics backbone does not by itself establish end-to-end VideoLLM efficiency.

Prioritized research agenda. 1) Establish a common analytical protocol. The most urgent need is a reproducible accuracy–compute protocol: candidate methods processing the same videos, prompts and modality inputs under fixed resolution and decoding settings, plug-in methods additionally sharing a frozen host, candidate-frame pool and frame or token budget. The protocol should report encoder, connector and LLM prefilling FLOPs under fixed accounting boundaries, including the cost of selection or allocation itself, together with retained-token counts and task performance both on the same input and under the same compute budget. FLOPs provide a hardware-independent common denominator without standing in for deployment speed; latency, memory and energy remain useful deployment measurements, but meaningful comparison requires a fixed hardware–software stack and measurement boundary [189, 190], and aggregating values from different stacks would create false precision. The emerging LLaVA-OneVision-7B, 32-frame, LMMs-Eval setup [226, 142, 126, 13, 135] is a practical starting point.

2) Report across video domains and task families. Efficiency results are reported almost exclusively as one aggregate accuracy on multiple-choice QA. Video-MME [38] annotates content domain as well as duration, yet most surveyed methods report only the aggregate, so a token budget validated on static lecture footage is indistinguishable from one validated on fast-cut sports. No comparison table in this survey contains a captioning, retrieval or grounding metric, and the one selector evaluated on captioning [91] had to be excluded from Table II for that reason. Multiple-choice questions supply the candidate answers and can often be settled by coarse object and scene cues; captions and temporal boundaries must be produced from finer detail. A retention ratio that is lossless on MCQ therefore need not be lossless on generation. Reporting per domain and re-testing one budget on a generation or localization task would check both assumptions cheaply.

3) Learn when audio should influence compression. Omni-modal models process synchronized audio and video [20, 72, 73], but the efficiency literature remains predominantly visual. OmniZip [15] and DASH [132] show that audio can guide visual-token reduction, yet their audio anchor may help in one segment and mislead in another: speech may refer to an off-screen event, while a visible event may have no informative sound. A stronger direction is a learned, query- and context-dependent allocation across modalities, evaluated on visual-only, audio-only, jointly answerable and deliberately conflicting examples, with the audio encoder and allocation module included in the cost.

VI Conclusion

This survey organized efficiency mechanisms for VideoLLMs by the stage of the encoder–connector–LLM pipeline at which they act: input construction and selection, encoder computation, encoded representations and connector, and LLM execution and state. Efficiency emerges from system-level trade-offs among semantic performance, input coverage, compute, latency and memory. Across the heterogeneous evidence reviewed here, retaining roughly one quarter of the visual-token budget often preserves near-baseline accuracy, though the achievable reduction depends on the host model, task and evaluation protocol. Consistent with prior pipeline analyses [25, 26], reducing LLM prefilling and cache costs can make vision encoding the limiting stage. In video systems, saved compute can also be reinvested in processing more frames, so gains must be interpreted together with temporal coverage and the cost of encoding those frames. Progress now requires a reproducible accuracy–compute protocol with common backbones, inputs and FLOP-accounting boundaries, with methods compared both on the same inputs and under the same compute budget and complemented by system measurements on a shared reference stack. Without it, reported gains remain difficult to compare across papers and to reproduce on a deployment target. Among mechanism directions, learned audiovisual allocation is especially promising because audio is widely available in video but still weakly represented in the efficiency literature.

References

• [1] Z. Tong, Y. Song, J. Wang, and L. Wang (2022) VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In Proc. NeurIPS, External Links: Document Cited by: §I.
• [2] Y. Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu, A. Vosoughi, C. Huang, Z. Zhang, P. Liu, M. Feng, F. Zheng, J. Zhang, P. Luo, J. Luo, and C. Xu (2026) Video Understanding with Large Language Models: A Survey. IEEE Transactions on Circuits and Systems for Video Technology 36 (2), pp. 1355–1376. External Links: Document Cited by: §I, §I, §III-A, §III-B.
• [3] N. Madan, A. Moegelmose, R. Modi, Y. S. Rawat, and T. B. Moeslund (2024) Foundation Models for Video Understanding: A Survey. Note: arXiv preprint arXiv:2405.03770 Cited by: §I, §I.
• [4] T. Nguyen, Y. Bin, J. Xiao, L. Qu, Y. Li, J. Z. Wu, C. Nguyen, S. Ng, and L. A. Tuan (2024) Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives. In Findings of the Association for Computational Linguistics ACL 2024, Cited by: §I, §I, §III-B.
• [5] M. A. Farag, M. H. Khafagy, and S. A. Hussien (2026) Video Captioning using Deep Learning with Greedy Search (VCDLGS). Franklin Open 14, pp. 100497. External Links: Document, ISSN 2773-1863 Cited by: §I.
• [6] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen (2024) A Survey on Multimodal Large Language Models. National Science Review 11 (12), pp. nwae403. External Links: Document, ISSN 2095-5138, 2053-714X Cited by: §I, §I.
• [7] Y. Weng, M. Han, H. He, X. Chang, and B. Zhuang (2024) LongVLM: Efficient Long Video Understanding via Large Language Models. In Proc. ECCV, External Links: Document Cited by: §I, §III-A.
• [8] D. Chatterjee, E. Remelli, Y. Song, B. Tekin, A. Mittal, B. Bhatnagar, N. C. Camgöz, S. Hampali, E. Sauser, S. Ma, A. Yao, and F. Sener (2025) Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding. In Proc. ICCV, External Links: Document Cited by: §I, §IV-A.
• [9] Z. Ning, G. Liu, Q. Jin, C. Li, W. Ding, M. Guo, and J. Zhao (2025) LiveVLM: efficient online video understanding via streaming-oriented kv cache and retrieval. Note: arXiv preprint arXiv:2505.15269 External Links: Document Cited by: §I.
• [10] S. Bhardwaj, M. Srinivasan, and M. M. Khapra (2019) Efficient Video Classification Using Fewer Frames. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, pp. 354–363. External Links: Document, ISBN 978-1-7281-3293-8 Cited by: §I.
• [11] X. Tang, J. Qiu, L. Xie, Y. Tian, J. Jiao, and Q. Ye (2025) Adaptive Keyframe Sampling for Long Video Understanding. In Proc. CVPR, External Links: Document Cited by: §I, Fig. 4, §IV-B1, §IV-B1, TABLE II.
• [12] X. Huang, H. Zhou, and K. Han (2025) PruneVid: visual token pruning for efficient video large language models. In Findings of ACL, pp. 19959–19973. External Links: Document Cited by: §I, Fig. 4, Fig. 4, §IV-D1, §IV-D5, §IV-E1, TABLE IV, TABLE V, TABLE V, §V.
• [13] J. Li, M. Wu, J. Cao, A. Tiulpin, and M. B. Blaschko (2026) EchoPrune: interpreting redundancy as temporal echoes for efficient VideoLLMs. arXiv preprint arXiv:2605.10050. Cited by: §I, Fig. 4, §IV-D1, §IV-D5, TABLE IV, §V, §V.
• [14] K. Tao, H. You, Y. Sui, C. Qin, and H. Wang (2025) Plug-and-play 1.x-bit KV cache quantization for video large language models. arXiv preprint arXiv:2503.16257. Cited by: §I, Fig. 4, §IV-E4.
• [15] K. Tao, K. Shao, B. Yu, W. Wang, J. liu, and H. Wang (2025) OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models. Note: arXiv preprint arXiv:2511.14582 External Links: Document Cited by: §I, Fig. 4, §IV-D1, TABLE IV, §V.
• [16] K. Tao, W. Du, B. Yu, W. Wang, J. Liu, and H. Wang (2025) OmniAgent: Audio-Guided Active Perception Agent for Omnimodal Audio-Video Understanding. Note: arXiv preprint arXiv:2512.23646 External Links: Document Cited by: §I.
• [17] H. Zou, T. Luo, G. Xie, Victor, Zhang, F. Lv, G. Wang, J. Chen, Z. Wang, H. Zhang, and H. Zhang (2024) From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding. Note: arXiv preprint arXiv:2409.18938 Cited by: §I, §III-B, §III-B.
• [18] J. Wu, W. Liu, Y. Liu, M. Liu, L. Nie, Z. Lin, and C. W. Chen (2026) A Survey on Video Temporal Grounding with Multimodal Large Language Model. IEEE Transactions on Pattern Analysis and Machine Intelligence. External Links: Document Cited by: §I.
• [19] Y. Kumar (2025) VideoLLM Benchmarks and Evaluation: A Survey. Note: arXiv preprint arXiv:2505.03829 External Links: Document Cited by: §I, §I.
• [20] L. Chen, J. Mu, J. Wang, X. Kang, X. Xi, and Z. Qin (2025) A survey on omni-modal language models. AI+. External Links: Document, ISSN 3007-7443, 3007-7451 Cited by: §I, §III-B, §V.
• [21] T. Bai, H. Liang, B. Wan, Y. Xu, X. Li, S. Li, L. Yang, B. Li, Y. Wang, B. Cui, P. Huang, J. Shan, C. He, B. Yuan, and W. Zhang (2024) A Survey of Multimodal Large Language Model from A Data-centric Perspective. Note: arXiv preprint arXiv:2405.16640 External Links: Document Cited by: §I.
• [22] D. Caffagni, F. Cocchi, L. Barsellotti, N. Moratelli, S. Sarto, L. Baraldi, L. Baraldi, M. Cornia, and R. Cucchiara (2024) The Revolution of Multimodal Large Language Models: A Survey. In Findings of ACL, Cited by: §I.
• [23] Y. Jin, J. Li, Y. Liu, T. Gu, K. Wu, Z. Jiang, M. He, B. Zhao, X. Tan, Z. Gan, Y. Wang, C. Wang, and L. Ma (2024) Efficient Multimodal Large Language Models: A Survey. Note: arXiv preprint arXiv:2405.10739 External Links: Document Cited by: §I, §IV-A.
• [24] K. Shao, K. Tao, K. Zhang, S. Feng, M. Cai, Y. Shang, H. You, C. Qin, Y. Sui, and H. Wang (2025) When Tokens Talk Too Much: A Survey of Multimodal Long-Context Token Compression across Images, Videos, and Audios. Note: arXiv preprint arXiv:2507.20198 External Links: Document Cited by: §I.
• [25] J. Zhang, Y. Ji, F. Ren, Y. Li, B. Zeng, Z. Chen, K. Chen, L. Shou, G. Chen, and H. Li (2026) Efficient inference for large vision-language models: bottlenecks, techniques, and prospects. In Findings of ACL, pp. 21036–21066. External Links: Document, Link Cited by: §I, §IV-A, §V, §VI.
• [26] H. Wu, J. Tong, X. Wang, Y. Tan, C. Zeng, A. Antsiferova, and X. Shen (2026) From data to model: a survey of the compression lifecycle in MLLMs. Note: TechRxiv preprint External Links: Document, Link Cited by: §I, §IV-A, §V, §VI.
• [27] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman (2017) The Kinetics Human Action Video Dataset. Note: arXiv preprint arXiv:1705.06950 External Links: Document Cited by: §II-B, TABLE III, §V.
• [28] R. Goyal, S. E. Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic (2017) The ”something something” video database for learning and evaluating visual common sense. In Proc. ICCV, External Links: Document Cited by: §II-B.
• [29] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik (2022) Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proc. CVPR, Cited by: §II-B.
• [30] J. Xu, T. Mei, T. Yao, and Y. Rui (2016) MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 5288–5296. External Links: Document, ISBN 978-1-4673-8851-1 Cited by: §II-B.
• [31] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles (2017) Dense-Captioning Events in Videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 706–715. External Links: Document Cited by: §II-B.
• [32] Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao (2019) ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering. Proc. AAAI. External Links: Document Cited by: §II-B.
• [33] J. Xiao, X. Shang, A. Yao, and T. Chua (2021) NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions. In Proc. CVPR, External Links: Document Cited by: §II-B.
• [34] K. Mangalam, R. Akshulakov, and J. Malik (2023) EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. In Proc. NeurIPS, External Links: Document Cited by: §II-B, §II-B, TABLE I, §III-A, §IV-A.
• [35] A. Miech, D. Zhukov, J. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic (2019) HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In Proc. ICCV, External Links: Document Cited by: §II-B.
• [36] J. Gao, C. Sun, Z. Yang, and R. Nevatia (2017) TALL: Temporal Activity Localization via Language Query. In Proc. ICCV, External Links: Document Cited by: §II-B.
• [37] K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao (2024) MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. In Proc. CVPR, External Links: Document Cited by: §II-B, TABLE I, §III-A, §III-A.
• [38] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun (2025) Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In Proc. CVPR, External Links: Document Cited by: §II-B, TABLE I, §III-A, §IV-B1, §V.
• [39] H. Wu, D. Li, B. Chen, and J. Li (2024) LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. In Proc. NeurIPS, Cited by: §II-B, TABLE I, §III-A, §IV-B3.
• [40] J. Zhou, Y. Shu, B. Zhao, B. Wu, Z. Liang, S. Xiao, M. Qin, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu (2025) MLVU: Benchmarking Multi-task Long Video Understanding. In Proc. CVPR, External Links: Document Cited by: TABLE I.
• [41] H. Zhang, Y. Wang, Y. Tang, Y. Liu, J. Feng, and X. Jin (2025) Flash-VStream: efficient real-time understanding for long video streams. In Proc. ICCV, Cited by: TABLE I, Fig. 4, §IV-D4, §IV-E6, TABLE IV, TABLE VI, TABLE VI.
• [42] H. Zhang, X. Li, and L. Bing (2023) Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, External Links: Document Cited by: §III-A, §III-A, Fig. 4, §IV-D3.
• [43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning Transferable Visual Models From Natural Language Supervision. In Proc. ICML, Cited by: §III-A, §IV-B1, §IV-C1.
• [44] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In Proc. ICLR, Cited by: §III-A.
• [45] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra (2023) ImageBind: One Embedding Space To Bind Them All. In Proc. CVPR, External Links: Document Cited by: §III-A.
• [46] J. Li, D. Li, S. Savarese, and S. Hoi (2023) BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proc. ICML, Cited by: §III-A, §III-B, §IV-A.
• [47] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Proc. NeurIPS, Vol. 36, pp. 46595–46623. Cited by: §III-A.
• [48] K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao (2025) VideoChat: Chat-Centric Video Understanding. Science China Information Sciences. External Links: Document Cited by: §III-A.
• [49] R. Luo, Z. Zhao, M. Yang, Z. Yang, M. Qiu, T. Wang, Z. Wei, Y. Wang, and C. Chen (2025) Valley: Video Assistant with Large Language model Enhanced abilitY. Note: arXiv preprint arXiv:2306.07207 External Links: Document Cited by: §III-A.
• [50] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan (2024) Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proc. ACL, External Links: Document Cited by: §III-A.
• [51] H. Xu, Q. Ye, X. Wu, M. Yan, Y. Miao, J. Ye, G. Xu, A. Hu, Y. Shi, G. Xu, C. Li, Q. Qian, M. Que, J. Zhang, X. Zeng, and F. Huang (2023) Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks. Note: arXiv preprint arXiv:2306.04362 Cited by: §III-A.
• [52] H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, C. Li, B. Bi, Q. Qian, W. Wang, G. Xu, J. Zhang, S. Huang, F. Huang, and J. Zhou (2023) mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video. In Proc. ICML, Cited by: §III-A.
• [53] Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li (2025) LLaVA-Video: Video Instruction Tuning With Synthetic Data. Note: arXiv preprint arXiv:2410.02713 External Links: Document Cited by: §III-A, §IV-B3, §IV-E6.
• [54] B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan (2024) Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Cited by: §III-A.
• [55] Y. Li, C. Wang, and J. Jia (2024) LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. In Proc. ECCV, External Links: Document Cited by: §III-A, Fig. 4, Fig. 4, §IV-D3, TABLE IV.
• [56] K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, D. Zhu, J. Ding, and M. Elhoseiny (2024) MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens. Note: arXiv preprint arXiv:2404.03413 Cited by: §III-A.
• [57] K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, M. Zhuge, J. Ding, D. Zhu, J. Schmidhuber, and M. Elhoseiny (2024) Goldfish: Vision-Language Understanding of Arbitrarily Long Videos. In Proc. ECCV, External Links: Document Cited by: §III-A, §III-A.
• [58] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin (2024) Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. Note: arXiv preprint arXiv:2409.12191 External Links: Document Cited by: §III-A.
• [59] Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y. Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y. Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang (2025) Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. Note: arXiv preprint arXiv:2412.05271 External Links: Document Cited by: §III-A, Fig. 4, §IV-D2, TABLE IV, §V.
• [60] Y. Wang, K. Li, X. Li, J. Yu, Y. He, C. Wang, G. Chen, B. Pei, Z. Yan, R. Zheng, J. Xu, Z. Wang, Y. Shi, T. Jiang, S. Li, H. Zhang, Y. Huang, Y. Qiao, Y. Wang, and L. Wang (2024) InternVideo2: Scaling Foundation Models for Multimodal Video Understanding. In Proc. ECCV, External Links: Document Cited by: §III-A.
• [61] Y. Wang, X. Li, Z. Yan, Y. He, J. Yu, X. Zeng, C. Wang, C. Ma, H. Huang, J. Gao, M. Dou, K. Chen, W. Wang, Y. Qiao, Y. Wang, and L. Wang (2025) InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling. Note: arXiv preprint arXiv:2501.12386 External Links: Document Cited by: §III-A.
• [62] E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang, Y. Lu, J. Hwang, and G. Wang (2024) MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. In Proc. CVPR, External Links: Document Cited by: §III-A, Fig. 4, §IV-D4, TABLE IV.
• [63] B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S. Lim (2024) MA-LMM: memory-augmented large multimodal model for long-term video understanding. In Proc. CVPR, Cited by: §III-A, Fig. 4, §IV-D4, TABLE IV.
• [64] R. Qian, X. Dong, P. Zhang, Y. Zang, S. Ding, D. Lin, and J. Wang (2024) Streaming Long Video Understanding with Large Language Models. In Proc. NeurIPS, Cited by: §III-A.
• [65] J. Chen, Z. Lv, S. Wu, K. Q. Lin, C. Song, D. Gao, J. Liu, Z. Gao, D. Mao, and M. Z. Shou (2024) VideoLLM-online: Online Video Large Language Model for Streaming Video. In Proc. CVPR, External Links: Document Cited by: §III-A.
• [66] S. Wu, J. Chen, K. Q. Lin, Q. Wang, Y. Gao, Q. Xu, T. Xu, Y. Hu, E. Chen, and M. Z. Shou (2024) VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation. In Proc. NeurIPS, External Links: Document Cited by: §III-A.
• [67] S. Santos, A. Farinhas, D. C. McNamee, and A. F. T. Martins (2025) $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation. In Proc. ICML, Cited by: §III-A, Fig. 4, Fig. 4, §IV-D4, TABLE IV.
• [68] X. Zeng, K. Li, C. Wang, X. Li, T. Jiang, Z. Yan, S. Li, Y. Shi, Z. Yue, Y. Wang, Y. Wang, Y. Qiao, and L. Wang (2025) TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning. In Proc. ICLR, pp. 38057–38081. Cited by: §III-A.
• [69] X. Li, Z. Yan, D. Meng, L. Dong, X. Zeng, Y. He, Y. Wang, Y. Qiao, Y. Wang, and L. Wang (2025) VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning. Note: arXiv preprint arXiv:2504.06958 Cited by: §III-A.
• [70] B. Chen, Z. Wang, Z. Yue, K. Yan, C. Yu, Y. Huang, Z. Liu, Y. Wen, X. Chen, Y. Liu, P. Li, and Y. Wang (2026) VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning. In Proc. CVPR, Cited by: §III-A.
• [71] Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao, and L. Bing (2024) VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. Note: arXiv preprint arXiv:2406.07476 Cited by: §III-A, Fig. 4, §IV-D2, TABLE IV.
• [72] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin (2025) Qwen2.5-Omni Technical Report. Note: arXiv preprint arXiv:2503.20215 External Links: Document Cited by: §III-A, §V.
• [73] H. Ye, C. H. Yang, A. Goel, W. Huang, L. Zhu, Y. Su, S. Lin, A. Cheng, Z. Wan, J. Tian, Y. Lou, D. Yang, Z. Liu, Y. Chen, A. Dantrey, E. Jahangiri, S. Ghosh, D. Xu, E. Hosseini-Asl, D. M. Taheri, V. Murali, S. Liu, Y. Lu, O. Olabiyi, Y. F. Wang, R. Valle, B. Catanzaro, A. Tao, S. Han, J. Kautz, H. Yin, and P. Molchanov (2025) OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM. Note: arXiv preprint arXiv:2510.15870 External Links: Document Cited by: §III-A, §V.
• [74] Z. Han, Q. Lin, H. Liang, B. Chen, Z. Liu, and W. Zhang (2026) LongInsightBench: A Comprehensive Benchmark for Evaluating Omni-Modal Models on Human-Centric Long-Video Understanding. In Findings of ACL, pp. 19332–19358. Cited by: §III-B.
• [75] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with io-awareness. In Proc. NeurIPS, NeurIPS 2022, pp. 16344–16359. External Links: Document Cited by: §III-B.
• [76] Z. Wu, C. Xiong, C. Ma, R. Socher, and L. S. Davis (2019) AdaFrame: Adaptive Frame Selection for Fast Video Recognition. In Proc. CVPR, Cited by: Fig. 4, §IV-B1, §IV-B1.
• [77] X. Zhang, Z. Wu, Z. Li, H. Xu, L. Gong, F. Boussaid, N. Werghi, and M. Bennamoun (2025) AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understanding. Note: arXiv preprint arXiv:2510.02778 External Links: Document Cited by: Fig. 4, §IV-B1, TABLE II.
• [78] S. Liu, C. Zhao, T. Xu, and B. Ghanem (2025) BOLT: boost large vision-language model without training for long-form video understanding. In Proc. CVPR, Cited by: Fig. 4, §IV-B1.
• [79] W. Chen, Y. Luo, Y. Zeng, L. Lin, T. Xie, F. Chao, R. Ji, and X. Zheng (2026) Event-anchored frame selection for effective long-video understanding. arXiv preprint arXiv:2603.00983. Cited by: Fig. 4, §IV-B1, TABLE II.
• [80] G. Sun, A. Singhal, B. Uzkent, M. Shah, C. Chen, and G. Kessler (2025) From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding. Note: arXiv preprint arXiv:2510.02262 External Links: Document Cited by: Fig. 4, Fig. 4, §IV-B1, §IV-B2.
• [81] Z. Zhu, H. Xu, Y. Luo, Y. Liu, K. Sarkar, Z. Yang, and Y. You (2025) FOCUS: Efficient Keyframe Selection for Long Video Understanding. Note: arXiv preprint arXiv:2510.27280 External Links: Document Cited by: Fig. 4, §IV-B1, TABLE II.
• [82] L. Yao, H. Wu, K. Ouyang, Y. Zhang, C. Xiong, B. Chen, X. Sun, and J. Li (2025) Generative Frame Sampler for Long Video Understanding. In Findings of ACL, External Links: Document Cited by: Fig. 4, §IV-B1.
• [83] J. Ma, S. Zhou, G. Li, X. Gao, Y. Cao, H. Zeng, Y. Yan, Z. Wang, J. Song, B. Zheng, S. Zhang, and J. Pu (2026) GIFT: global irreplaceability frame targeting for efficient video understanding. In Proc. CVPR, Cited by: Fig. 4, §IV-B1, §IV-B3.
• [84] Y. Yang and K. Lam (2025) HFS: Holistic Query-Aware Frame Selection for Efficient Video Reasoning. Note: arXiv preprint arXiv:2512.11534 External Links: Document Cited by: Fig. 4, §IV-B1.
• [85] M. Afham, S. N. Shukla, O. Poursaeed, P. Zhang, A. Shah, and S. Lim (2023) Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understanding. In Proc. ICCV, Cited by: Fig. 4, §IV-B1.
• [86] B. Song, J. Peng, Y. Zhang, G. Chen, F. Yang, and J. Guo (2026) KTV: keyframes and key tokens selection for efficient training-free video LLMs. In Proc. AAAI, Cited by: Fig. 4, Fig. 4, §IV-B1, §IV-D1.
• [87] J. Chen, J. Qian, W. Deng, Y. Guo, J. Yu, S. Leng, R. Thirukovalluru, and B. Dhingra (2026) LDDR: linear-DPP-based dynamic-resolution frame sampling for video MLLMs. arXiv preprint arXiv:2605.11477. Cited by: Fig. 4, Fig. 4, §IV-B1, §IV-B2.
• [88] X. Shen, Y. Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, Z. Liu, H. Xu, H. J. Kim, B. Soran, R. Krishnamoorthi, M. Elhoseiny, and V. Chandra (2025) LongVU: spatiotemporal adaptive compression for long video-language understanding. In Proc. ICML, Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-D1, TABLE IV.
• [89] P. Li, I. Abdullaeva, A. Gambashidze, A. Kuznetsov, and I. Oseledets (2026) MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding. In Proc. WACV, External Links: Document Cited by: Fig. 4, §IV-B1, TABLE II.
• [90] Y. Zhi, Z. Tong, L. Wang, and G. Wu (2021) MGSampler: An Explainable Sampling Strategy for Video Action Recognition. In Proc. ICCV, External Links: Document Cited by: Fig. 4, §IV-B1.
• [91] K. Steunou, A. Filali Razzouki, K. Guetari, M. A. El-Yacoubi, and Y. Tevissen (2026) PEEK: picking essential frames via efficient knowledge distillation. In British Machine Vision Conference (BMVC), Cited by: Fig. 4, §IV-B1, §IV-B3, §V.
• [92] S. Zhang, J. Yang, J. Yin, Z. Luo, and J. Luan (2025) Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs. In Proc. ICCV, External Links: Document Cited by: Fig. 4, Fig. 4, §IV-B1, §IV-B2.
• [93] J. Peng, B. Song, J. Li, H. Li, Y. Zhou, R. Ji, and Y. Tian (2026) QCA: query- and content-aware keyframe selection for long video understanding. In Proc. ECCV, Cited by: Fig. 4, §IV-B1, TABLE II.
• [94] Y. Qin, H. Li, W. Mu, and Y. He (2026) Efficient frame selection for long video understanding via reinforcement learning. In Proc. CVPR, pp. 16944–16953. Cited by: Fig. 4, §IV-B1.
• [95] H. Lee, J. Kim, H. Kim, and Y. M. Ro (2026) ReFoCUS: reinforcement-guided frame optimization for contextual understanding. In Proc. CVPR, Cited by: Fig. 4, §IV-B1.
• [96] J. Ye, Z. Wang, H. Sun, K. Chandrasegaran, Z. Durante, C. Eyzaguirre, Y. Bisk, J. C. Niebles, E. Adeli, L. Fei-Fei, J. Wu, and M. Li (2025) T*: re-thinking temporal search for long-form video understanding. In Proc. CVPR, Cited by: Fig. 4, §IV-B1.
• [97] T. Qu, M. Li, T. Tuytelaars, and M. Moens (2024) TS-LLaVA: constructing visual tokens through thumbnail-and-sampling for training-free video large language models. Note: arXiv preprint arXiv:2411.11066 Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-B2, §IV-D2, TABLE IV.
• [98] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool (2016) Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. In Proc. ECCV, External Links: Document Cited by: Fig. 4, §IV-B1.
• [99] C. Tang, Z. Han, H. Sun, S. Zhou, X. Zhang, X. Wei, Y. Yuan, H. Zhang, J. Xu, and H. Sun (2026) TSPO: temporal sampling policy optimization for long-form video language understanding. In Proc. AAAI, Cited by: Fig. 4, §IV-B1, §IV-B3, TABLE II.
• [100] Z. Xu, Q. Dai, T. Xie, Y. Yang, K. Qiu, D. Chen, Z. Wu, and C. Luo (2025) ViaRL: adaptive temporal grounding via visual iterated amplification reinforcement learning. arXiv preprint arXiv:2505.15447. Cited by: Fig. 4, §IV-B1.
• [101] S. Wang, G. Chen, D. Huang, Z. Li, M. Li, G. Liu, J. M. Alvarez, L. Zhang, and Z. Yu (2025) VideoITG: multimodal video understanding with instructed temporal grounding. arXiv preprint arXiv:2507.13353. Cited by: Fig. 4, §IV-B1, §IV-B3.
• [102] B. Shi, S. Fu, L. Lian, H. Ye, D. Eigen, A. Reite, B. Li, J. Kautz, S. Han, D. M. Chan, P. Molchanov, T. Darrell, and H. Yin (2026) Attend before attention: efficient and scalable video understanding via autoregressive gazing. In Proc. CVPR, Cited by: Fig. 4, §IV-B2.
• [103] Z. Wu, S. Wang, and X. Yu (2026) MeToM: metadata-guided token merging for efficient video LLMs. In Proc. CVPR, pp. 10441–10448. Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-B2, §IV-D1, §IV-E1, §V.
• [104] H. Akbari, L. Yuan, R. Qian, W. Chuang, S. Chang, Y. Cui, and B. Gong (2021) VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text. In Proc. NeurIPS, Cited by: Fig. 4, §IV-B2, TABLE III.
• [105] P. K. A. Vasu, F. Faghri, C. Li, C. Koc, N. True, A. Antony, G. Santhanam, J. Gabriel, P. Grasch, O. Tuzel, and H. Pouransari (2025) FastVLM: efficient vision encoding for vision language models. In Proc. CVPR, Cited by: Fig. 4, §IV-C1.
• [106] C. Ryali, Y. Hu, D. Bolya, C. Wei, H. Fan, P. Huang, V. Aggarwal, A. Chowdhury, O. Poursaeed, J. Hoffman, J. Malik, Y. Li, and C. Feichtenhofer (2023) Hiera: A hierarchical vision transformer without the bells-and-whistles. In Proc. ICML, Cited by: Fig. 4, §IV-C1, §IV-C4, TABLE III.
• [107] J. Kim, N. Parthasarathy, D. Qin, J. Hur, D. Sun, B. Han, M. Yang, and B. Gong (2026) LiteFrame: efficient vision encoders unlock frame scaling in video LLMs. Note: arXiv preprint arXiv:2605.17260 Cited by: Fig. 4, §IV-C1.
• [108] J. Alayrac, A. Recasens, R. Schneider, R. Arandjelović, J. Ramapuram, J. D. Fauw, L. Smaira, S. Dieleman, and A. Zisserman (2020) Self-Supervised MultiModal Versatile Networks. In Proc. NeurIPS, Cited by: Fig. 4, §IV-C1, TABLE III.
• [109] P. K. A. Vasu, H. Pouransari, F. Faghri, R. Vemulapalli, and O. Tuzel (2024) MobileCLIP: fast image-text models through multi-modal reinforced training. In Proc. CVPR, Cited by: Fig. 4, §IV-C1.
• [110] F. Faghri, P. K. A. Vasu, C. Koc, V. Shankar, A. Toshev, O. Tuzel, and H. Pouransari (2025) MobileCLIP2: improving multi-modal reinforced training. Transactions on Machine Learning Research. Cited by: Fig. 4, §IV-C1.
• [111] M. Yang, Z. Jia, Z. Dai, S. Guo, and L. Wang (2025) MobileViCLIP: An Efficient Video-Text Model for Mobile Devices. In Proc. ICCV, External Links: Document Cited by: Fig. 4, §IV-C1.
• [112] B. Zhang, S. Dong, Q. H. Tran, K. Gschwind, S. Yang, S. Chen, A. Ahmadyan, S. Moon, L. Zhang, A. Kirmani, B. Damavandi, and A. Kumar (2026) MoE-ViE: mixture of experts vision encoder for efficient image and video understanding. In Proc. ECCV, Cited by: Fig. 4, §IV-C1.
• [113] D. Kondratyuk, L. Yuan, Y. Li, L. Zhang, M. Tan, M. Brown, and B. Gong (2021) MoViNets: mobile video networks for efficient video recognition. In Proc. CVPR, pp. 16020–16030. External Links: Document Cited by: Fig. 4, Fig. 4, §IV-C1, §IV-C3, TABLE III.
• [114] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer (2021) Multiscale Vision Transformers. In Proc. ICCV, External Links: Document Cited by: Fig. 4, §IV-A, §IV-C1, TABLE III.
• [115] Y. Li, C. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer (2022) MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. In Proc. CVPR, External Links: Document Cited by: Fig. 4, §IV-C1, §IV-C4, TABLE III.
• [116] Z. Liu, Y. Dong, Z. Liu, W. Hu, J. Lu, and Y. Rao (2025) Oryx MLLM: on-demand spatial-temporal understanding at arbitrary resolution. In Proc. ICLR, Cited by: Fig. 4, Fig. 4, §IV-C1, §IV-D3, TABLE IV.
• [117] K. Wu, H. Peng, Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. Chen, X. Wang, H. Chao, and H. Hu (2023) TinyCLIP: CLIP distillation via affinity mimicking and weight inheritance. In Proc. ICCV, Cited by: Fig. 4, §IV-C1.
• [118] J. Lin, C. Gan, and S. Han (2019) TSM: Temporal Shift Module for Efficient Video Understanding. In Proc. ICCV, External Links: Document Cited by: Fig. 4, Fig. 4, §IV-C1, §IV-C3, TABLE III.
• [119] K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao (2022) UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning. In Proc. ICLR, Cited by: Fig. 4, §IV-C1, TABLE III.
• [120] K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, L. Wang, and Y. Qiao (2023) UniFormerV2: spatiotemporal learning by arming image ViTs with video UniFormer. In Proc. ICCV, Cited by: Fig. 4, §IV-C1, §IV-C4, TABLE III.
• [121] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu (2022) Video Swin Transformer. In Proc. CVPR, Cited by: Fig. 4, §IV-C1, §IV-C4, TABLE III.
• [122] K. Li, X. Li, Y. Wang, Y. He, Y. Wang, L. Wang, and Y. Qiao (2024) VideoMamba: state space model for efficient video understanding. In Proc. ECCV, pp. 237–255. External Links: Document Cited by: Fig. 4, §IV-C1, TABLE III.
• [123] J. Park, H. Kim, K. Ko, M. Kim, and C. Kim (2024) VideoMamba: spatio-temporal selective state space model. In Proc. ECCV, External Links: Document Cited by: Fig. 4, §IV-C1, TABLE III.
• [124] H. Lu, A. A. Salah, and R. Poppe (2025) Snakes and ladders: two steps up for VideoMamba. In Proc. ICCV, Cited by: Fig. 4, §IV-C1, §IV-C4, TABLE III, §V.
• [125] C. Feichtenhofer (2020) X3D: expanding architectures for efficient video recognition. In Proc. CVPR, pp. 203–213. External Links: Document Cited by: Fig. 4, §IV-A, §IV-C1, TABLE III.
• [126] H. Wang, X. Jin, L. Lu, C. Li, J. Chen, Q. Liu, and H. Wang (2026) EarlyTom: early token compression completes fast video understanding. Proc. CVPR. Cited by: Fig. 4, Fig. 4, §IV-A, §IV-C2, §IV-D5, TABLE V, TABLE V, §V, §V.
• [127] M. Soldan, F. Caba Heilbron, B. Ghanem, J. Sivic, and B. Russell (2025) ResidualViT for efficient temporally dense video encoding. In Proc. ICCV, Cited by: Fig. 4, Fig. 4, §IV-C2, §IV-C3.
• [128] D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman (2023) Token merging: your ViT but faster. In Proc. ICLR, Cited by: Fig. 4, §IV-C2.
• [129] S. D. Sarkar, R. Pautrat, O. Miksik, M. Pollefeys, I. Armeni, M. Rad, and M. Dusmanu (2026) CoPE-VideoLM: leveraging codec primitives for efficient video language modeling. arXiv preprint arXiv:2602.13191. Cited by: Fig. 4, §IV-C3, §V.
• [130] Y. Wang, X. Liu, X. Gui, X. Lin, B. Yang, C. Liao, T. Chen, and L. Zhang (2026) Accelerating streaming video large language models via hierarchical token compression. Proc. CVPR. Cited by: Fig. 4, Fig. 4, §IV-C3, §V.
• [131] P. Jin, R. Takanobu, W. Zhang, X. Cao, and L. Yuan (2024) Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. In Proc. CVPR, External Links: Document Cited by: Fig. 4, §IV-D1, TABLE IV.
• [132] B. Li and T. Huang (2026) DASH: dynamic audio-driven semantic chunking for efficient omnimodal token compression. Proc. ECCV. Cited by: Fig. 4, §IV-D1, TABLE IV, §V, §V.
• [133] K. Tao, C. Qin, H. You, Y. Sui, and H. Wang (2025) DyCoke: dynamic compression of tokens for fast video large language models. In Proc. CVPR, pp. 18992–19001. Cited by: Fig. 4, Fig. 4, §IV-D1, §IV-D5, §IV-E4, TABLE V, §V.
• [134] M. Park, T. Kong, and S. Ahn (2026) DynaTok: temporally adaptive and positional bias-aware token compression for Video-LLMs. Note: arXiv preprint arXiv:2605.19322 Cited by: Fig. 4, §IV-D1.
• [135] Y. Li, H. Gui, Z. Fan, J. Wang, B. Kang, B. Chen, and Z. Tian (2025) Less is more, but where? dynamic token compression via LLM-guided keyframe prior. Proc. NeurIPS. Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-D1, §V, §V.
• [136] L. Shen, G. Gong, T. He, Y. Zhang, P. Liu, S. Zhao, and G. Ding (2025) FastVID: dynamic density pruning for fast video large language models. In Proc. NeurIPS, Cited by: Fig. 4, §IV-D1, TABLE IV, TABLE V, TABLE V.
• [137] Z. Fan, K. Chen, R. Xing, Y. Li, L. Jiang, and Z. Tian (2026) FlashVID: efficient video large language models via training-free tree-based spatiotemporal token merging. Proc. ICLR. Cited by: Fig. 4, Fig. 4, §IV-D1, §IV-D5, §IV-E1, TABLE IV, TABLE V, TABLE V, §V, §V.
• [138] Y. Zhang, Y. Lu, T. Wang, F. Rao, Y. Yang, and L. Zhu (2025) FlexSelect: flexible token selection for efficient long video understanding. In Proc. NeurIPS, Cited by: Fig. 4, §IV-D1, §V.
• [139] S. Ju, B. Song, T. Chen, J. Zhang, Q. Wu, C. Chang, H. Wang, Y. Zhou, and R. Ji (2026) ForestPrune: high-ratio visual token compression for video multimodal large language models via spatial-temporal forest modeling. In Proc. CVPR, Cited by: Fig. 4, §IV-D1.
• [140] S. Yu, C. Jin, H. Wang, Z. Chen, S. Jin, Z. Zuo, X. Xu, Z. Sun, B. Zhang, J. Wu, H. Zhang, and Q. Sun (2025) Frame-Voyager: Learning to Query Frames for Video Large Language Models. Note: arXiv preprint arXiv:2410.03226 External Links: Document Cited by: Fig. 4, §IV-A, §IV-D1, §V.
• [141] Y. Guo, C. Zhu, J. Ji, J. Lin, and L. Cao (2026) HieraVid: hierarchical token pruning for fast video large language models. arXiv preprint arXiv:2604.01881. Cited by: Fig. 4, Fig. 4, §IV-A, §IV-E1, §IV-E6, TABLE VII, TABLE VII, §V, §V.
• [142] K. Shao, K. Tao, C. Qin, H. You, Y. Sui, and H. Wang (2025) HoliTom: holistic token merging for fast video large language models. In Proc. NeurIPS, Cited by: Fig. 4, Fig. 4, §IV-A, §IV-D1, §IV-D5, §IV-E1, TABLE IV, TABLE V, TABLE V, TABLE V, §V, §V, §V.
• [143] X. Liu, S. Gan, X. Liu, Y. Yin, L. Xie, and S. Lu (2026) InfoMerge: information-aware token compression for efficient video large language models. Note: arXiv preprint arXiv:2606.02161 Cited by: Fig. 4, §IV-D1.
• [144] Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan (2025) LLaVA-PruMerge: adaptive token reduction for efficient large multimodal models. In Proc. ICCV, pp. 22857–22867. Cited by: Fig. 4, §IV-D1, TABLE IV, §V.
• [145] B. Sun, J. Zhao, X. Wei, and Q. Hou (2025) LLaVA-Scissor: token compression with semantic connected components for video LLMs. Note: arXiv preprint arXiv:2506.21862 Cited by: Fig. 4, §IV-D1, TABLE IV.
• [146] J. Ma, Q. Zhang, M. Lu, Z. Wang, Q. Zhou, J. Song, and S. Zhang (2026) MMG-Vid: maximizing marginal gains at segment-level and token-level for efficient video LLMs. In Proc. AAAI, Cited by: Fig. 4, §IV-D1, TABLE IV, TABLE V.
• [147] M. Kang, M. Lee, J. Lee, M. Kim, D. Kim, D. Lee, H. Choi, I. Kim, and S. Lee (2026) OTT-Vid: optimal transport temporal token compression for video large language models. Note: arXiv preprint arXiv:2605.11803 Cited by: Fig. 4, §IV-D1.
• [148] J. Jiang, X. Li, Z. Liu, M. Li, G. Chen, Z. Li, D. Huang, G. Liu, Z. Yu, K. Keutzer, S. Ahn, J. Kautz, H. Yin, Y. Lu, S. Han, and W. Byeon (2025) STORM: token-efficient long video understanding for multimodal LLMs. In Proc. ICCV, pp. 5889–5900. External Links: Document Cited by: Fig. 4, Fig. 4, §IV-D2, TABLE IV, §V.
• [149] X. Chen, K. Tao, K. Shao, and H. Wang (2026) StreamingTOM: streaming token compression for efficient video understanding. In Proc. CVPR, Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-A, §IV-D1, §IV-E5, TABLE IV.
• [150] L. Yao, Y. Li, Y. Wei, L. Li, S. Ren, Y. Liu, K. Ouyang, L. Wang, S. Li, S. Li, L. Kong, Q. Liu, Y. Zhang, and X. Sun (2025) TimeChat-Online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia (ACM MM), Cited by: Fig. 4, §IV-D1, TABLE IV.
• [151] X. Liu, Y. Wang, J. Ma, and L. Zhang (2025) Video compression commander: plug-and-play inference acceleration for video large language models. In Proc. EMNLP, Cited by: Fig. 4, §IV-D1, TABLE IV, TABLE V, §V.
• [152] X. Li, Y. Wang, J. Yu, X. Zeng, Y. Zhu, H. Huang, J. Gao, K. Li, Y. He, C. Wang, Y. Qiao, Y. Wang, and L. Wang (2026) VideoChat-Flash: hierarchical compression for long-context video modeling. In Proc. ICLR, Cited by: Fig. 4, Fig. 4, §IV-D1, §IV-E1, TABLE IV.
• [153] S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia (2025) VisionZip: longer is better but not necessary in vision language models. In Proc. CVPR, pp. 19792–19802. External Links: Document Cited by: Fig. 4, §IV-D1, §IV-D5, TABLE IV, TABLE V, TABLE V, §V.
• [154] Y. Li, H. Sun, M. Lin, T. Li, G. Dong, T. Zhang, B. Ding, W. Song, Z. Cheng, Y. Huo, S. Chen, X. Li, D. Pan, S. Zhang, X. Wu, Z. Liang, J. Liu, T. Zhang, K. Lu, Y. Zhao, Y. Shen, F. Yang, K. Yu, T. Lin, J. Xu, Z. Zhou, and W. Chen (2024) Baichuan-Omni Technical Report. Note: arXiv preprint arXiv:2410.08565 External Links: Document Cited by: Fig. 4, §IV-D2, TABLE IV.
• [155] H. X. Team and N. Cloud (2026) HyperCLOVA X 8B Omni. Note: arXiv preprint arXiv:2601.01792 External Links: Document Cited by: Fig. 4, §IV-D2, TABLE IV.
• [156] Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y. Lou, S. Yang, H. Xi, S. Cao, Y. Gu, D. Li, X. Li, Y. Fang, Y. Chen, C. Hsieh, D. Huang, A. Cheng, V. Nath, J. Hu, S. Liu, R. Krishna, D. Xu, X. Wang, P. Molchanov, J. Kautz, H. Yin, S. Han, and Y. Lu (2025) NVILA: efficient frontier visual language models. In Proc. CVPR, Cited by: Fig. 4, §IV-D2, TABLE IV.
• [157] L. Xu, Y. Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng (2024) PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning. Note: arXiv preprint arXiv:2404.16994 Cited by: Fig. 4, §IV-D2, TABLE IV, §V.
• [158] C. Yang, X. Dong, X. Zhu, W. Su, J. Wang, H. Tian, Z. Chen, W. Wang, L. Lu, and J. Dai (2025) PVC: progressive visual token compression for unified image and video processing in large vision-language models. In Proc. CVPR, Cited by: Fig. 4, §IV-D2, TABLE IV.
• [159] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou (2024) Qwen2-Audio technical report. Note: arXiv preprint arXiv:2407.10759 Cited by: Fig. 4, §IV-D2.
• [160] M. Xu, M. Gao, Z. Gan, H. Chen, Z. Lai, H. Gang, K. Kang, and A. Dehghan (2024) SlowFast-LLaVA: a strong training-free baseline for video large language models. Note: arXiv preprint arXiv:2407.15841 Cited by: Fig. 4, §IV-D2, TABLE IV.
• [161] X. Lan, Y. Yuan, Z. Jie, and L. Ma (2024) VidCompress: memory-enhanced temporal compression for video understanding in large language models. Note: arXiv preprint arXiv:2410.11417 Cited by: Fig. 4, Fig. 4, Fig. 4, §IV-D3, §IV-D4, TABLE IV.
• [162] R. Li, Y. Tan, Y. Shi, and J. Shao (2025) VideoScan: enabling efficient streaming video understanding via frame-level semantic carriers. Note: arXiv preprint arXiv:2503.09387 Cited by: Fig. 4, Fig. 4, §IV-D2, §IV-E4, TABLE IV.
• [163] M. S. Ryoo, H. Zhou, S. Kendre, C. Qin, L. Xue, M. Shu, J. Park, K. Ranasinghe, S. Savarese, R. Xu, C. Xiong, and J. C. Niebles (2024) xGen-MM-Vid (BLIP-3-Video): you only need 32 tokens to represent a video even in VLMs. Note: arXiv preprint arXiv:2410.16267 Cited by: Fig. 4, Fig. 4, §IV-D3, §IV-D4, TABLE IV.
• [164] G. Sun, W. Yu, C. Tang, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang (2023) Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models. Note: arXiv preprint arXiv:2310.05863 Cited by: Fig. 4, §IV-D3, TABLE IV.
• [165] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan (2022) Flamingo: a Visual Language Model for Few-Shot Learning. In Proc. NeurIPS, Cited by: Fig. 4, §IV-A, §IV-D3.
• [166] S. Zhang, Q. Fang, Z. Yang, and Y. Feng (2025) LLaVA-Mini: efficient image and video large multimodal models with one vision token. In Proc. ICLR, Cited by: Fig. 4, §IV-D3, TABLE IV.
• [167] J. Qi, Y. Yao, Y. Bai, B. Xu, J. Li, Z. Liu, and T. Chua (2025) An LMM for efficient video understanding via reinforced compression of video cubes. Note: arXiv preprint arXiv:2504.15270 Cited by: Fig. 4, §IV-D3, TABLE IV.
• [168] G. Sun, W. Yu, C. Tang, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, Y. Wang, and C. Zhang (2024) Video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models. In Proc. ICML, pp. 47198–47217. Cited by: Fig. 4, §IV-D3, TABLE IV.
• [169] H. Zhang and Y. Fu (2025) VQToken: neural discrete token representation learning for extreme token reduction in video large language models. In Proc. NeurIPS, Cited by: Fig. 4, §IV-D3.
• [170] Y. Man, Y. Huang, C. Zhang, B. Li, W. Niu, and M. Yin (2025) AdaCM${}^2$: on understanding extremely long-term video with adaptive cross-modality memory reduction. In Proc. CVPR, Cited by: Fig. 4, §IV-D4.
• [171] Y. Wang, Y. Song, C. Xie, Y. Liu, and Z. Zheng (2025) VideoLLaMB: long streaming video understanding with recurrent memory bridges. In Proc. ICCV, Cited by: Fig. 4, §IV-D4, TABLE IV.
• [172] F. Sun, L. Shen, H. Chen, S. Zhao, J. Han, and G. Ding (2025) AdaTP: attention-debiased token pruning for video large language models. In Findings of EMNLP, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, pp. 3273–3286. External Links: Document, ISBN 979-8-89176-335-7 Cited by: Fig. 4, §IV-E1.
• [173] L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang (2024) An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In Proc. ECCV, Cited by: Fig. 4, §IV-D1, §IV-E1, §IV-E6, TABLE VII, §V.
• [174] T. Fu, T. Liu, Q. Han, G. Dai, S. Yan, H. Yang, X. Ning, and Y. Wang (2025) FrameFusion: combining similarity and importance for video token reduction on large vision language models. In Proc. ICCV, Cited by: Fig. 4, §IV-E1, §IV-E6, TABLE VII, §V.
• [175] J. Jiang, A. S. Deshmukh, K. Chumachenko, K. Sapra, Z. Yu, G. Liu, A. Tao, P. Molchanov, J. Kautz, and W. Byeon (2026) Stateful token reduction for long-video hybrid VLMs. arXiv preprint arXiv:2603.00198. Cited by: Fig. 4, §IV-E1, §V, §V.
• [176] Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, and S. Zhang (2025) SparseVLM: visual token sparsification for efficient vision-language model inference. In Proc. ICML, Cited by: Fig. 4, §IV-E1, §V.
• [177] J. Hyun, S. Hwang, S. H. Han, T. Kim, I. Lee, D. Wee, J. Lee, S. J. Kim, and M. Shim (2025) Multi-granular spatio-temporal token merging for training-free acceleration of video LLMs. In Proc. ICCV, Cited by: Fig. 4, §IV-E1, TABLE IV.
• [178] B. Xu, Z. Xiao, J. Li, J. Ju, Z. Luo, J. Luan, and Q. Jin (2026) TimeViper: a hybrid mamba-transformer vision-language model for efficient long video understanding. Proc. CVPR. Cited by: Fig. 4, §IV-E1, §V.
• [179] M. Kim, K. Shim, J. Choi, and S. Chang (2025) InfiniPot-V: memory-constrained KV cache compression for streaming video understanding. In Proc. NeurIPS, Cited by: Fig. 4, §IV-E4, §IV-E6, TABLE VI, TABLE VI.
• [180] Z. Wan, H. Shen, X. Wang, C. Liu, Z. Mai, and M. Zhang (2025) MEDA: dynamic KV cache allocation for efficient multimodal long-context inference. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: Fig. 4, §IV-E4.
• [181] X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nie (2024) ReTaKe: reducing temporal and knowledge redundancy for long video understanding. Note: arXiv preprint arXiv:2412.20504 Cited by: Fig. 4, §IV-E4, TABLE VI, §V.
• [182] Y. Chen, X. Bai, Z. Wang, C. Bai, Y. Dai, M. Lu, and S. Zhang (2026) StreamKV: streaming video question-answering with segment-based KV cache retrieval and compression. In Proc. AAAI, Cited by: Fig. 4, Fig. 4, §IV-E5.
• [183] Y. Yang, Z. Zhao, S. N. Shukla, A. Singh, S. K. Mishra, L. Zhang, and M. Ren (2025) StreamMem: query-agnostic KV cache memory for streaming video understanding. arXiv preprint arXiv:2508.15717. Cited by: Fig. 4, §IV-E4, §IV-E6, TABLE VI, TABLE VI, TABLE VI.
• [184] Y. Li, H. Jiang, C. Zhang, Q. Wu, X. Luo, S. Ahn, A. H. Abdi, D. Li, J. Gao, Y. Yang, and L. Qiu (2025) MMInference: accelerating pre-filling for long-context VLMs via modality-aware permutation sparse attention. In Proc. ICML, Cited by: Fig. 4, §IV-E2, §IV-E6.
• [185] S. Di, Z. Yu, G. Zhang, H. Li, T. Zhong, H. Cheng, B. Li, W. He, F. Shu, and H. Jiang (2025) Streaming video question-answering with in-context video KV-cache retrieval. In Proc. ICLR, Cited by: Fig. 4, Fig. 4, §IV-E2, §IV-E5, §IV-E6, TABLE VI.
• [186] Y. Shu, Z. Liu, P. Zhang, M. Qin, J. Zhou, Z. Liang, T. Huang, and B. Zhao (2025) Video-XL: extra-long vision language model for hour-scale video understanding. In Proc. CVPR, Cited by: Fig. 4, §IV-E3, TABLE IV.
• [187] X. Ye, Y. Gan, X. Huang, Y. Ge, Y. Shan, and Y. Tang (2025) VoCo-LLaMA: towards vision compression with large language models. In Proc. CVPR, pp. 29836–29846. Cited by: Fig. 4, §IV-E3, TABLE IV.
• [188] M. Dehghani, A. Arnab, L. Beyer, A. Vaswani, and Y. Tay (2022) The efficiency misnomer. In Proc. ICLR, Cited by: §IV-A.
• [189] V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. S. John, P. Kanwar, D. Lee, J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhimenko, A. T. R. Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou (2020) MLPerf Inference Benchmark. In Proc. ISCA, Cited by: §IV-A, §V.
• [190] A. Tschand, A. T. R. Rajan, S. Idgunji, A. Ghosh, J. Holleman, C. Kiraly, P. Ambalkar, R. Borkar, R. Chukka, T. Cockrell, O. Curtis, G. Fursin, M. Hodak, H. Kassa, A. Lokhmotov, D. Miskovic, Y. Pan, M. P. Manmathan, L. Raymond, T. S. John, A. Suresh, R. Taubitz, S. Zhan, S. Wasson, D. Kanter, and V. J. Reddi (2025) MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), External Links: Document Cited by: §IV-A, §V.
• [191] J. Liu, Y. Wang, L. Zhang, Y. Wang, S. Xu, L. Wang, J. Yan, D. Zhang, and X. Chen (2025) Towards training-free long video understanding: methods, benchmarks, and open challenges. Vicinagearth 2 (1), pp. 6. External Links: Document, ISSN 3005-060X Cited by: §IV-A.
• [192] M. Brkic, A. F. Razzouki, Y. Tevissen, K. Guetari, and M. A. E. Yacoubi (2025) Frame Sampling Strategies Matter: A Benchmark for small vision language models. Note: arXiv preprint arXiv:2509.14769 External Links: Document Cited by: §IV-B1.
• [193] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu (2021) Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling. In Proc. CVPR, External Links: Document Cited by: §IV-B1.
• [194] D. Potapov, M. Douze, Z. Harchaoui, and C. Schmid (2014) Category-Specific Video Summarization. In Proc. ECCV, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.), Cham, pp. 540–555. External Links: Document, ISBN 978-3-319-10599-4 Cited by: §IV-B1.
• [195] S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei (2016) End-to-end Learning of Action Detection from Frame Glimpses in Videos. In Proc. CVPR, External Links: Document Cited by: §IV-B1.
• [196] H. Fan, Z. Xu, L. Zhu, C. Yan, J. Ge, and Y. Yang (2018) Watching a Small Portion could be as Good as Watching All: Towards Efficient Video Classification. In Proc. IJCAI, Cited by: §IV-B1.
• [197] W. Wu, D. He, X. Tan, S. Chen, and S. Wen (2019) Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition. In Proc. ICCV, Cited by: §IV-B1.
• [198] Z. Wu, C. Xiong, Y. Jiang, and L. S. Davis (2019) LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition. In Proc. NeurIPS, Cited by: §IV-B1.
• [199] Y. Meng, C. Lin, R. Panda, P. Sattigeri, L. Karlinsky, A. Oliva, K. Saenko, and R. Feris (2020) AR-Net: Adaptive Frame Resolution for Efficient Action Recognition. In Proc. ECCV, Cited by: §IV-B1.
• [200] H. Liang, J. Li, T. Bai, C. Chen, C. He, B. Cui, and W. Zhang (2024) KeyVideoLLM: Towards Large-scale Video Keyframe Selection. Note: arXiv preprint arXiv:2407.03104 Cited by: §IV-B1.
• [201] H. Sun, S. Lu, H. Wang, Q. Chen, Z. Xu, W. Luo, K. Zhang, and M. Li (2025) MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs. In Proc. ICCV, External Links: Document Cited by: §IV-B1.
• [202] W. Guo, Z. Chen, S. Wang, J. He, Y. Xu, J. Ye, Y. Sun, and H. Xiong (2025) Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding. In Proc. NeurIPS, External Links: Document Cited by: §IV-B1.
• [203] M. Chasmai, G. Jagatap, G. KV, G. V. Horn, S. Maji, and A. Fanelli (2025) Moment Sampling in Video LLMs for Long-Form Video QA. Note: arXiv preprint arXiv:2507.00033 External Links: Document Cited by: §IV-B1.
• [204] E. Jang, S. Gu, and B. Poole (2017) Categorical reparameterization with gumbel-softmax. In Proc. ICLR, Cited by: §IV-B1.
• [205] K. Hu, F. Gao, X. Nie, P. Zhou, S. Tran, T. Neiman, L. Wang, M. Shah, R. Hamid, B. Yin, and T. Chilimbi (2025) M-LLM Based Video Frame Selection for Efficient Video Understanding. In Proc. CVPR, External Links: Document Cited by: §IV-B1.
• [206] Y. Yao, Y. Yun, J. Wang, H. Zhang, D. Zhao, K. Tian, Z. Wang, M. Qiu, and T. Wang (2025) K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding. Note: arXiv preprint arXiv:2510.13891 External Links: Document Cited by: §IV-B1.
• [207] C. Li, T. Li, F. Tao, Z. Zhao, Z. Wu, M. Zhao, J. Song, C. Niu, and P. Fazli (2025) FrameOracle: Learning What to See and How Much to See in Videos. Note: arXiv preprint arXiv:2510.03584 External Links: Document Cited by: §IV-B1.
• [208] Z. Li, K. Ishida, S. Yamazaki, X. Ji, and J. Liu (2026) KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding. In Proc. WACV, External Links: Document Cited by: §IV-B3.
• [209] A. Gu and T. Dao (2024) Mamba: linear-time sequence modeling with selective state spaces. In Proc. COLM, Cited by: §IV-C1.
• [210] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li (2025) LLaVA-onevision: easy visual task transfer. Transactions on Machine Learning Research. Cited by: §IV-C1, §IV-D5, §IV-E4.
• [211] M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, S. A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfruend, C. Vondrick, and A. Oliva (2020) Moments in Time Dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. External Links: Document Cited by: TABLE III, §V.
• [212] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter (2017) Audio Set: an ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776–780. Cited by: TABLE III.
• [213] K. Soomro, A. R. Zamir, and M. Shah (2012) UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv preprint arXiv:1212.0402. Cited by: TABLE III, §V.
• [214] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre (2011) HMDB: a large video database for human motion recognition. In Proc. ICCV, pp. 2556–2563. Cited by: TABLE III.
• [215] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid Loss for Language Image Pre-Training. In Proc. ICCV, Cited by: §IV-D1.
• [216] Y. Kumar (2025) Language-guided temporal token pruning for efficient VideoLLM processing. In Proc. EMNLP, Cited by: §IV-D1.
• [217] H. Zhang, J. Zhang, X. Ji, Q. Wang, and F. Zhang (2025) DynTok: dynamic compression of visual tokens for efficient and effective video understanding. Note: arXiv preprint arXiv:2506.03990 Cited by: §IV-D1.
• [218] B. Lenz, O. Lieber, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y. Belinkov, S. Shalev-Shwartz, O. Abend, R. Alon, T. Asida, A. Bergman, R. Glozman, M. Gokhman, A. Manevich, N. Ratner, N. Rozen, E. Shwartz, M. Zusman, and Y. Shoham (2025) Jamba: a hybrid transformer-mamba language model. In Proc. ICLR, Cited by: §IV-E1, §V.
• [219] X. Liu, Y. Shu, Z. Liu, A. Li, Y. Tian, and B. Zhao (2025) Video-XL-Pro: reconstructive token compression for extremely long video understanding. Note: arXiv preprint arXiv:2503.18478 Cited by: §IV-E3.
• [220] M. Qin, X. Liu, Z. Liang, Y. Shu, H. Yuan, J. Zhou, S. Xiao, B. Zhao, and Z. Liu (2025) Video-XL-2: towards very long-video understanding through task-aware KV sparsification. Note: arXiv preprint arXiv:2506.19225 Cited by: §IV-E3.
• [221] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. Note: arXiv preprint arXiv:2502.13923 Cited by: §IV-E4.
• [222] X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nie (2025) AdaReTaKe: adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503.12559. Cited by: §IV-E4.
• [223] Z. Lin, M. Lin, L. Lin, and R. Ji (2025) Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proc. AAAI, Cited by: §IV-E4.
• [224] L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin (2025) PyramidDrop: accelerating your large vision-language models via pyramid visual redundancy reduction. In Proc. CVPR, Cited by: §IV-E4.
• [225] Z. Wan, Z. Wu, C. Liu, J. Huang, Z. Zhu, P. Jin, L. Wang, and L. Yuan (2024) LOOK-M: look-once optimization in KV cache for efficient multimodal long-context inference. In Findings of EMNLP, pp. 4065–4078. External Links: Document Cited by: §IV-E4.
• [226] K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, and Z. Liu (2025) LMMs-eval: reality check on the evaluation of large multimodal models. In Findings of NAACL, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, pp. 881–916. External Links: Document, ISBN 979-8-89176-195-7 Cited by: §V.