# Adobe 用 Python 函数替代固定参考答案做智能体评测

- 来源：DAIR.AI (@dair_ai)
- 发布时间：2026-09-17 04:00
- AIHOT 分数：43
- AIHOT 链接：https://aihot.news/items/cmu4k0emg0tsoro4wfbs90wo5
- 原文链接：https://x.com/dair_ai/status/2100313810011832555

## AI 摘要

Adobe 研究者提出把每条评测用例的参考答案写成 Python 函数，评测时对实时系统运行，让期望答案随数据变化，上游 API 变更会直接导致测试失败。

## 正文

Nice work from Adobe.

It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale.

Adobe researchers write each reference answer as a Python function instead.

The function runs against the live system at evaluation time, so the expected answer follows the data, and an upstream API change makes the test fail visibly.

An LLM judge then splits the agent's response and the computed answer into atomic facts and scores precision and recall, whatever the output format.

Against expert labels, this raises agreement from an MCC of 0.331 to 0.427 and cuts token cost per case by 16%. A judge working with no ground truth scored an MCC of -0.379, which is worse than chance.

The pipeline runs as a harness skill. The authors list one limitation, which is that the same model, Claude Sonnet 4.6, acted as both the skill and the judge.

Paper: https://academy.dair.ai/papers/skill-based-agentic-evaluation-for-real-time-data-science-tasks-2609.16487