苹果研究：LLM 做机器翻译质量标注，人类与模型都面临挑战
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
苹果研究团队评估了 LLM 在 MQM 和 ESA 两种机器翻译质量标注方案下与人类标注者的一致性，测试覆盖 70 个语言对的长上下文测试集及 WMT23、WMT25 公开数据。结果显示，LLM 在某些任务上与人类的一致性超过人类标注者之间的一致性，但在多数标注方案、语言对和领域下仍不可靠。人类在细粒度 MQM 标注和低资源语言对上尤为吃力，LLM 则难以处理细微错误、错误语言变体和错误跨度标注。
AuthorsHala Almaghout, Christian Federmann, Qin Gao
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.
Related readings and updates.
Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these…
Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and…
来源：Apple Machine Learning Research · machinelearning.apple.com