AutomationBench Verified 发布，对 Zapier AutomationBench 全部 600 个公开任务的评分器逐一审计，Agent 标记 323 个可疑项，人工确认 206 个真实 bug 并全部修复。
Agent benchmarks need audits of the graders, not only the tasks.
Parsewave reviewed all 600 public tasks in Zapier's AutomationBench, which frontier labs cite on their model cards.
It used agents to write convincing wrong answers, then had humans confirm which graders actually failed: 206 did.
AutomationBench Verified fixes every one.
On 1,235 Kimi K3 runs, the fixed graders gave a different verdict 27.9% of the time.
The clearest case is task 813. The grader checked the Salesforce notes but never looked at the DocuSign template. A submission that sent four contracts on the Standard template, instead of the required GDPR, HIPAA, SOC2 and Enterprise ones, passed 13 of 13 checks and scored 1.0. After the fix it scores 0.09.
AutomationBench Verified is out. We went through all 600 public tasks in @Zapier's AutomationBench and checked every verifier. • Agents flagged 323 verifiers as suspicious, human review confirmed 206 real bugs, all 206 are fixed. • We replayed 1,235 Kimi K3 runs on the old and fixed verifiers and 344 of them (27.9%) got a different grade • Where verifiers were too strict, pass rate went from 18.8% to 43.8% • Where they were too lenient, it dropped from 60.2% to 49.7% Audit and dataset links below.在 X 查看被引用的帖子
来源：Rohan Paul · x.com