# 论文主张将校准作为 LLM 评估的一等标准

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-22 08:00
- AIHOT 分数：36
- AIHOT 链接：https://aihot.news/items/cmufdsjtd07juroagmmdk8cex
- 原文链接：https://arxiv.org/abs/2609.26489

## AI 摘要

一篇 NLP 论文主张把校准（模型置信度与经验正确率的一致性）列为 LLM 评估的一等标准，而非小众子领域。作者指出，LLM-as-a-judge、合成数据生成和主动学习等方法都依赖校准置信度却未加验证，而现有大多数 benchmark 已同时提供置信度分数与正确性判断，校准可立即报告；开放生成场景下这两个输入的界定仍是未解难题。

## 正文

Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the research pipeline, where methods like LLM-as-a-judge, synthetic data generation, and active learning rely on calibrated confidence without verifying it.

Standard calibration metrics only require two inputs per example: a confidence score and a correctness judgment. Most benchmarks in use today already provide both, meaning calibration can be reported immediately. For open-ended generation, however, defining these two inputs is still an open challenge. We argue that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic.