Rohan Paul· @rohanpaul_ai · X·· 2026-07-05AI 评分55
AI 导读
耶鲁大学和芝加哥大学最新论文基于11,683篇真实论文构造对照测试,让LLM基于与人类相同的先前工作提出研究动机和方法。结果发现,人类研究者想法模式多样(如解释机制、测试失败、测量证据、构建系统、提升效率),仅12.1%属于“连接已有工作”类;而LLM生成的同类想法占比高达47.1%至64.2%,频率约为人类的4至5倍。即使增加推理步骤(额外CoT),这一连接偏好反而更强,说明LLM倾向于优化已有配方,而非探索多样化的研究路径。
正文 · 原文
This is the prompt Yale and Univ of Chicago researchers used when asking LLMs for new research ideas.
Feed LLMs prior work, ask for ideas, then measure how repetitive the ideas get.
The surprising finding is that LLMs often treat research ideation as connecting what already exists, while humans use a wider set of problem-finding moves.
LLM-generated ideas reveal a bias toward safe bridge-and-combine proposals.
This Yale + University of Chicago paper shows that real gap between LLM generated research ideas vs humans is not idea quality, but idea range: LLMs think narrower than human researchers. The researchers built a controlled test from 11,683 real papers, using each paper’s nearby prior work as the shared starting point. They asked models to propose a new motivation and method from those same prior papers, then compared those ideas with the real human paper ideas. Instead of asking whether 1 idea looked novel, they labeled each idea by what gap it noticed and what kind of contribution it made. Human ideas spread across many patterns, such as explaining mechanisms, testing failures, measuring evidence, building systems, and improving efficiency. Only 12.1% of human ideas were mainly about connecting separate work, but 47.1% to 64.2% of LLM ideas did that, meaning models used this move about 4 to 5 times more often. Even extra reasoning made this pattern stronger, suggesting models often polish a familiar recipe instead of finding more varied research moves. --- – arxiv. org/abs/2607.01233 Title: "Measuring the Gap Between Human and LLM Research Ideas"在 X 查看被引用的帖子
来源:Rohan Paul · x.com