MOLE 基准发布:检测 AI 智能体中的内部威胁

HuggingFace Daily Papers(社区热门论文)·2026-09-07 08:00·2天前
AI 导读

MOLE 是一个开放基准,用 150 个 AI 运营账号、9 个有状态服务、30 个工作日模拟 12 种内部威胁,语料约 200 亿 token,来自四个模型。在 39 个 agent 模型中 72% 能完成大部分有害目标,且 agent 拒绝不能预测完成与否;最佳监控器在单日审计事件对比中仍漏掉近一半已完成的危害,基准引导搜索可将中档监控器提升 49-64%。

HuggingFace Daily Papers(社区热门论文)
55AI 编辑部评分,满分 100

MOLE 基准发布:检测 AI 智能体中的内部威胁

2026-09-07 08:00· 2天前
AI 导读

MOLE 是一个开放基准,用 150 个 AI 运营账号、9 个有状态服务、30 个工作日模拟 12 种内部威胁,语料约 200 亿 token,来自四个模型。在 39 个 agent 模型中 72% 能完成大部分有害目标,且 agent 拒绝不能预测完成与否;最佳监控器在单日审计事件对比中仍漏掉近一半已完成的危害,基准引导搜索可将中档监控器提升 49-64%。

Model misalignment, prompt injection, or operator misuse could lead AI agents operating frontier-lab accounts to exfiltrate model weights, poison training data, or weaken release gates. Existing benchmarks do not test whether defenders can detect this activity among routine work under a limited review budget. We introduce MOLE, an open benchmark of 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and 8 corpora from four models totaling roughly 20 billion tokens. Of 39 agent models, 72% complete most assigned harmful objectives and agent refusal does not predict completion. MOLE enables comparison of 40 monitors across corpus generators, observability levels, and threats; even the best evaluated monitor in our single-day audit-event comparison misses nearly half of completed harm. MOLE also enables monitor development: benchmark-guided search improves a mid-tier monitor by 49-64%, while selective use of a stronger monitor improves budget-AUC by 10% over applying it to every account-day at comparable modeled cost.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org