A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.
IdeaAMBIG:评测研究想法规格中阻碍实现的关键缺口
AI 导读
研究者提出 IdeaAMBIG 基准,包含 660 个有证据支撑的实例,其中 163 个来自复现报告和 GitHub issue 的真实缺口、497 个注入到可编码参考中的受控合成缺口,用于评测编码就绪度评估、缺陷定位和澄清动作生成三项能力。
HuggingFace Daily Papers(社区热门论文)
40
AI 编辑部评分,满分 100IdeaAMBIG:评测研究想法规格中阻碍实现的关键缺口
研究者提出 IdeaAMBIG 基准,包含 660 个有证据支撑的实例,其中 163 个来自复现报告和 GitHub issue 的真实缺口、497 个注入到可编码参考中的受控合成缺口,用于评测编码就绪度评估、缺陷定位和澄清动作生成三项能力。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org