尽管多模态领域泛化（MMDG）在提升模型鲁棒性方面日益流行，但报告的性能提升究竟是反映了真正的算法进步，还是不一致评估协议带来的人为产物，目前仍不明确。当前研究较为零散，各研究在数据集、模态配置和实验设置上差异显著。此外，现有基准主要聚焦于动作识别，往往忽略了输入损坏、模态缺失和模型可信度等关键的现实世界挑战。
这种标准化的缺失使得对该领域进展的可靠评估变得模糊不清。为解决这一问题，我们引入了 MMDG-Bench，这是首个统一且全面的 MMDG 基准，它标准化了涵盖三个不同任务（动作识别、机械故障诊断和情感分析）的六个数据集的评估。
MMDG-Bench 包含六种模态组合、九种代表性方法以及多种评估设置。除了标准准确率之外，它还系统性地评估了损坏鲁棒性、缺失模态泛化能力、误分类检测以及分布外检测。MMDG-Bench 总共训练了 7,402 个神经网络，跨越 95 个独特的跨领域任务，得出了五个关键发现：（1）在公平比较下，近期专门的 MMDG 方法相比 ERM 基线仅带来边际提升；（2）没有任何单一方法能在不同数据集或模态组合上持续优于其他方法；（3）与上限性能之间仍存在显著差距，表明 MMDG 远未得到解决；（4）三模态融合并不总是优于最强的双模态配置；（5）所有被评估的方法在损坏和模态缺失场景下均表现出显著性能下降，且部分方法进一步损害了模型可信度。
Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness.
This lack of standardization obscures a reliable assessment of the field's advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection.
With 7, 402 neural networks trained in total across 95 unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness.