# ImpossibleRubrics：用 169 个不可能任务压力测试生成式评分标准作为奖励信号

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-15 08:00
- AIHOT 分数：50
- AIHOT 链接：https://aihot.news/items/cmu426fov06k7ro4w10va8dnl
- 原文链接：https://arxiv.org/abs/2609.16816

## AI 摘要

研究者提出 ImpossibleRubrics 基准，包含 169 个不可能任务（覆盖六类不可能性）和 48 个可回答对照任务，每个任务配有可验证的 oracle certificate，用于检验下游生成的评分标准是否会奖励违反证书的答案。

## 正文

Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if they reward honest answers over adversarial answers optimized to exploit them. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, together with 48 answerable controls.

Rather than providing fixed rubrics, ImpossibleRubrics provides task environments and certificates, allowing rubrics to be generated downstream and then adversarially tested for whether they reward certificate-violating answers. Eleven generators are exploited 8--26% of the time on the unbiased 150-of-169 environment cut; on a deliberately selected stress cut the strongest generator we measured is still exploited 36% while a certificate-faithful rubric is exploited 0%, so what we measure is a rubric-quality gap, not task impossibility. One result runs against intuition.

A single generic rubric ("be decisive, penalize hedging") used unchanged for every task is exploited 64% of the time, and seven of the eleven generators are exploited more often than that while writing a rubric tailored to each one. The tailored criteria appear to tell an attacker which claim to fabricate. The problem is not that rubrics are vague; it is that they are specific about the wrong things.