用多项选择题问答评估视频描述质量：Apple 提出信息保真度新指标
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Apple 提出以信息保真度重新定义视频描述质量，认为描述需最大化对视频内容的覆盖。现有指标依赖生成文本与参考文本的匹配，因视频描述存在"一对多"特性，高质量描述常因用词差异或视觉重点合理偏移而被扣分，且评估维度单一，无法细粒度分析描述质量。
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the “one-to-many” nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. We further formulate CapF1, a composite metric that synthesizes CapP (measuring factuality) and CapR (measuring coverage). Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
- † MBZUAI Institute of Foundation Models
- ** Work done while at Apple
来源：Apple Machine Learning Research · machinelearning.apple.com