Abstract
AI research agents now support large-scale AI-assisted scientific discovery. We examine whether AI-generated ideas broaden scientific exploration or primarily reinforce existing work. Using five agent frameworks and five large language models, we generate 219,655 ideas for different scientific fields. Across experiments, four consistent patterns emerge. First, AI-generated ideas are more concentrated than human-authored papers within the same research area. Second, they remain much closer to starting literature than later human follow-on work does. Third, AI-generated ideas align less with future human research. Last, AI-generated ideas are located in lower-impact regions of the historical scientific landscape. Overall, current AI research agents appear better suited to local elaboration than to broadening scientific exploration.
keywords:
Artificial intelligence, Agentic AI, Scientific discovery, Large language models, Science of science
1 Introduction
Recent advances in AI research agents have raised the possibility of automating scientific discovery. These agents can now conduct literature reviews, generate research ideas, plan experiments, run code, write papers, and iteratively explore and refine scientific hypotheses [lu2026towards, baek2025researchagent, schmidgall2025agent, hao2026artificial]. Importantly, these AI research agent frameworks are explicitly designed to encourage exploratory scientific ideation. Their prompts and reasoning procedures often instruct agents to generate novel, high-impact, and unconventional ideas rather than simple extensions of prior work [lu2026towards, baek2025researchagent, schmidgall2025agent]. As such systems become increasingly capable and accessible, they may fundamentally reshape the process of scientific discovery.
Yet the ability to generate research ideas at scale does not necessarily imply broader scientific exploration. Scientific discovery often depends on moving beyond established directions, searching less familiar regions, and recombining prior knowledge in non-routine ways [foster2015tradition, uzzi2013atypical]. Existing evaluations of AI research agents mainly assess whether individual ideas are interesting, novel, feasible, or executable [si2024canllms, wang2024scimon], but reveal much less about how repeated AI-assisted ideation may shape the broader landscape of scientific exploration. This raises a broader question: do AI research agents broaden scientific exploration?
To study this question, we construct research areas from the scientific literature across broad fields and use AI research agents to generate ideas from shared seed papers within each research area. Specifically, using papers published between 2020 and 2025 from the Semantic Scholar Academic Graph as seed literature, we use advanced AI research-agent frameworks, including AIScientist [lu2026towards], ResearchAgent [baek2025researchagent], AgentLaboratory [schmidgall2025agent], and Co-Scientist [gottweis2026coscientist], together with five LLMs to generate complete scientific research ideas, including both research questions and methods. In total, we analyze 219,655 valid AI-generated research ideas spanning 12 broad scientific fields and 155 research areas. Throughout the ideation process, all evaluated AI agent frameworks are explicitly instructed to explore novel research directions beyond the seed literature. The AI agent frameworks can further search and retrieve additional relevant literature from the entire Semantic Scholar database.
We then investigate these AI-generated ideas from four perspectives: whether they explore diverse scientific directions, whether they move beyond their starting literature toward new research topics, whether they align with future research frontiers, and whether they are associated with potentially high-impact regions of the scientific landscape.
Across research fields and consistently across evaluated agent frameworks and underlying LLMs, four patterns emerge. First, AI-generated ideas are substantially more concentrated than human-authored papers from the same research areas. Second, AI-generated ideas remain much closer to their starting literature than later human follow-on work does, indicating that they primarily extrapolate locally from prior work. Third, AI-generated ideas cover substantially fewer keywords characterizing the next year’s research frontier, defined by the most frequently studied topics in subsequent human research, than follow-on human papers. Finally, AI-generated ideas are associated with human-authored papers that receive fewer citations than human follow-on work.
These findings suggest a cautious view of the role of current AI research agents in scientific discovery. Although current AI agents can efficiently generate coherent, literature-grounded research ideas at scale, their outputs do not appear to substantially expand the frontier of scientific exploration. As AI systems become increasingly integrated into scientific workflows, the distinction between scaling idea generation and broadening scientific exploration may become increasingly important. The broader challenge is therefore not simply to make AI systems generate more plausible scientific ideas, but to design systems that genuinely expand the range of scientific exploration.
| Framework | Agentic mechanism | Implementation summary | Novelty instruction excerpt |
|---|---|---|---|
| Zero-shot | Single-pass generation | The LLM receives literature context and generates a research idea in a single interaction. | “propose one novel research idea grounded in the literature” |
| AI Scientist [lu2026towards] | Iterative self-reflection | The agent iteratively critiques and revises generated ideas, optionally refreshing literature context between rounds. | “propose any novel ideas or experiments; make sure they are novel”; “quality, novelty, and feasibility” |
| ResearchAgent [baek2025researchagent] | Multi-stage planning and validation | The agent decomposes ideation into problem finding, method design and experiment planning, with intermediate validation agents scoring outputs. | “promising, new, and key scientific problems”; “original”; “innovative”; validation dimensions include “Originality” and “Innovativeness” |
| AgentLaboratory [schmidgall2025agent] | Multi-agent deliberation | Multiple role-based agents iteratively discuss and refine research proposals through dialogue. | “very innovative and unlike anything seen before”; “Make sure your new output is very different” |
| Co-Scientist [gottweis2026coscientist] | Tournament-based hypothesis evolution | Multiple agents generate competing hypotheses, then review, rank, and evolve them through tournament comparison before a final meta-review. | “develop one novel, feasible research hypothesis” and “rank hypotheses by novelty, significance, feasibility, and testability” |
The final column presents excerpts of explicit novelty-related instructions from each framework.
2 Generating Scientific Ideas with AI Agents
Define Research Areas.
We begin by constructing research areas from the scientific literature across major fields of science. We collect papers from the Semantic Scholar Academic Graph111https://www.semanticscholar.org/product/api, together with their reference and citation information. The resulting corpus spans 12 fields, including Medicine, Biology, Engineering, Chemistry, Computer Science, Environmental Science, Materials Science, Physics, Mathematics, Economics, Business, and Sociology. Each paper record includes its abstract, publication year, primary field, and citation links. We use papers published before 2020 to construct research areas.
Within each scientific field, we identify research areas using bibliographic coupling [kessler1963bibliographic]. Specifically, papers are represented by their citation profiles and clustered according to bibliographic-coupling similarity. The final identified research areas span topics including cryo-electron microscopy, pancreatic cancer treatment, offshore wind power, heavy-ion physics, and microbiome community assembly. Detailed construction procedures are provided in the Supplementary Methods (see SM S1.1).
Scientific Idea generation.
We next use AI research agents to generate new scientific ideas from prior literature. For each identified research area, We repeatedly sample seed-paper sets from papers published between 2020 and 2025 to initialize AI ideation. Each seed set contains five papers: one anchor paper together with four related papers from the same research area, selected using citation. We use five seed papers because most evaluated AI research agent frameworks are constrained by context-window limitations of current LLMs. These seed papers define a coherent research topic and provide a common starting literature context for idea generation.
During idea generation, the evaluated agent frameworks can further search a locally deployed Semantic Scholar database to retrieve additional relevant papers, allowing them to expand the literature context beyond the initial seed set. To preserve the historical setting, agents are only allowed to retrieve papers that were available when the seed papers were published.
We evaluate five representative AI research-agent frameworks: a Zero-shot baseline, AIScientist [lu2026towards], ResearchAgent [baek2025researchagent], AgentLaboratory [schmidgall2025agent], and Co-Scientist [gottweis2026coscientist]. These frameworks represent several major designs for AI research agents, including iterative self-reflection, multi-stage planning and validation, multi-agent deliberation, and tournament-based hypothesis evolution. All evaluated agent frameworks, except the Zero-shot baseline, can retrieve additional papers from the locally deployed Semantic Scholar database. Table 1 summarizes the evaluated frameworks and their corresponding agentic mechanisms.
Importantly, across all evaluated frameworks, the ideation prompts explicitly encourage exploration beyond the seed literature. The Zero-shot baseline asks the model to propose a novel research idea. AIScientist emphasizes generating “novel” and “high-impact” ideas through iterative self-reflection and revision. ResearchAgent encourages innovative method design during multi-stage planning and validation. Agent Laboratory explicitly instructs agents to expand upon the literature and generate ideas that are “very innovative and unlike anything seen before.” Co-Scientist generates and refines hypotheses through comparison, and explicitly rewards novelty at each round. Supplementary Methods provides the full prompts and detailed agentic design of each evaluated framework (see SM S1.2).
Each AI agent framework needs to be paired with an LLM. We evaluate five LLMs: four open-weight models ranging from 8B to 35B parameters, Gemma-4-31B-IT [farabet_lacombe_gemma4_2026], Llama-3.1-8B [grattafiori2024llama], Hermes-4-14B [teknium2025hermes], and Qwen3-35B-A3B [qwen3.5], together with OpenAI’s GPT-5.4 [openai_chat].
Across the five agent frameworks and five LLMs, we bootstrap seed-paper sets from the identified research areas. In total, our analysis uses 219,655 valid AI-generated ideas generated from 155 research areas spanning 12 broad scientific fields, obtained from 232,800 generation runs. A generation run is considered valid if it successfully produces a non-empty structured research idea (see SM S1.2, tables S3 and S4 for the detailed breakdown and examples).
3 Quantifying AI-Generated Scientific Ideas
We next introduce several measurements to characterize AI-generated scientific ideas.
Exploration breadth. We measure exploration breadth as the extent to which AI-generated ideas spread across distinct directions within an identified research area [peng2021neural]. To quantify this breadth, we encode every AI-generated idea into a shared semantic embedding space using a text embedding model. Exploration breadth is then measured as the average pairwise cosine distance among AI-generated ideas within the same research area. Higher average cosine distance indicates that generated ideas occupy a broader region of the semantic idea space and therefore explore a wider range of scientific directions [pnasHofstra2020, cohan2020specter]. As a robustness check, we also quantify exploration breadth using a centroid-based distance measure (see SM S2.2 and table S6).
Exploration distance. We measure exploration distance as the extent to which AI-generated ideas move beyond the seed literature used to initialize ideation [shibayama2021measuring]. For each idea, we first compute the centroid of the five human-authored seed papers in the semantic embedding space. Exploration distance is then quantified as the cosine distance between each AI-generated idea and the corresponding seed-paper centroid. Larger distances indicate that generated ideas move further away from their starting literature.
Frontier alignment. We measure frontier alignment as the extent to which AI-generated ideas align with emerging research directions. For each broad scientific field, we define the next-year field frontier as the set of the top 10% most frequent scholarly keywords extracted from human-authored papers published in the year following the seed-paper set [cui2026aging]. To obtain a common representation, we use an LLM to extract scholarly keywords from both AI-generated ideas and human-authored papers. For each identified research area, we aggregate the extracted keywords from all AI-generated ideas into a single AI keyword set. Frontier alignment is then computed as the proportion of frontier keywords that also appear in the aggregated AI keyword set. Higher frontier alignment indicates that AI-generated ideas align more closely with the future human research frontier.
Potential scientific impact. We measure the potential scientific impact of AI-generated ideas based on the citation performance of semantically similar human-authored papers [arts2025beyond]. Because AI-generated ideas themselves have no citation records, we use nearby human-authored papers in the semantic embedding space as observable proxies. We first compute a normalized citation score for each human-authored paper relative to papers published in the same research area and publication year. We then identify the 20 nearest human-authored papers for each AI-generated idea and define its potential impact score as the average normalized citation score of these neighbors. Higher impact scores indicate that AI-generated ideas are located in potentially higher-impact regions of the scientific landscape.
4 Empirical Analysis
4.1 Exploration Breadth: AI Ideas Are More Concentrated Than Human Papers
{tableorg}
[t] Exploration breadth by agent framework and LLM. Exploration breadth is first computed within each research area and then averaged across research areas. The AI and Human columns report the mean exploration breadth of AI-generated ideas and the corresponding human-authored papers, respectively. AI–Human denotes the difference between AI-generated ideas and human-authored papers. Brackets report 95% bootstrap confidence intervals. Group AI Human AI–Human [95% CI] Pooled All data 0.554 0.599 -0.045 [-0.049, -0.042] By agent framework Agent Laboratory 0.541 0.599 -0.058 [-0.063, -0.053] AIScientist 0.548 0.599 -0.051 [-0.055, -0.046] Co-Scientist 0.563 0.599 -0.036 [-0.040, -0.030] ResearchAgent 0.570 0.599 -0.029 [-0.033, -0.025] Zero-shot 0.546 0.599 -0.053 [-0.058, -0.048] By LLM GPT-5.4 0.518 0.585 -0.067 [-0.080, -0.060] Gemma-4-31B-IT 0.562 0.599 -0.037 [-0.041, -0.032] Hermes-4-14B 0.553 0.599 -0.047 [-0.051, -0.041] Llama-3.1-8B 0.545 0.599 -0.054 [-0.059, -0.050] Qwen3-35B-A3B 0.572 0.599 -0.027 [-0.032, -0.023]
We examine the exploration breadth of AI-generated ideas within each identified research area. As a comparison, we also measure the exploration breadth of the human-authored seed papers. To ensure a fair comparison, we randomly sample one seed paper for each AI-generated idea so that the human and AI collections contain the same number of ideas.
Figure 2a–b shows a consistent pattern across the five agent frameworks and five LLMs. AI-generated ideas within the same research area are more similar to one another than human-authored papers from those same areas. Averaged across agent frameworks, the breadth is 0.554 for AI-generated ideas and 0.599 for human-authored papers, representing a 7.5% lower exploration breadth for AI-generated ideas.
Panels c–d further show that different LLMs and agent frameworks often explore overlapping regions of the same research area. Within the same research area, the average exploration breadth between ideas generated by different agent frameworks is 0.572, while that between ideas generated by different LLMs is 0.570. These averages remain lower than the human same-area baseline. Field-level analyses show the same pattern across 11 of the 12 broad scientific fields, with Mathematics being the only exception (see SM S2.1 and table S5).
We observe the same pattern using an alternative centroid-based measure of exploration breadth. For each research area, AI-generated ideas lie closer to their area centroids than human-authored papers do, again indicating a more concentrated exploration pattern (see SM S2.2 and table S6).
4.2 Exploration Distance: AI Ideas Stay Close to Their Starting Literature
We next examine the exploration distance of AI-generated ideas to assess whether they move beyond the seed literature used to initialize ideation or instead remain locally anchored to it. As a comparison, we examine follow-on human-authored papers that directly cite at least one of the seed papers [wu2019large]. These follow-on papers represent subsequent human research emerging from the same research topic.
Figure 3 compares the distributions of exploration distance for AI-generated ideas and follow-on human papers across four consecutive years (20202021 through 20232024). Across all four years, AI-generated ideas remain closer to the seed literature than subsequent human research. In every year, the AI distributions are shifted toward smaller exploration distances, whereas follow-on human papers exhibit broader distributions extending to substantially larger distances. On average, across years, exploration distance is 0.322 for AI-generated ideas, compared with 0.410 for follow-on human papers. Field-level analyses reveal the same pattern across all twelve broad scientific fields, with the difference remaining statistically significant in every field (see SM S2.1 and table S5). This result suggests that, although all evaluated AI agent frameworks can search for and retrieve relevant literature from the Semantic Scholar database, AI-generated ideas remain largely confined to local exploration.
{tableorg}
[t] Exploration distance from seed literature by year, agent framework, and LLM. Exploration distance is first computed for each seed-paper set and then averaged across seed-paper sets. The AI and Human columns report the mean exploration distance of AI-generated ideas and the corresponding follow-on human papers, respectively. AI–Human denotes the difference between AI-generated ideas and follow-on human papers. Brackets report 95% bootstrap confidence intervals. Group AI Human AI–Human [95% CI] By year All years 0.322 0.410 -0.088 [-0.092, -0.083] 20202021 0.318 0.399 -0.081 [-0.090, -0.072] 20212022 0.320 0.407 -0.087 [-0.097, -0.077] 20222023 0.322 0.411 -0.089 [-0.098, -0.079] 20232024 0.328 0.422 -0.093 [-0.103, -0.084] By agent framework Agent Laboratory 0.382 0.410 -0.027 [-0.033, -0.022] AIScientist 0.320 0.410 -0.090 [-0.095, -0.085] Co-Scientist 0.328 0.410 -0.082 [-0.087, -0.077] ResearchAgent 0.313 0.410 -0.096 [-0.101, -0.091] Zero-shot 0.265 0.410 -0.144 [-0.149, -0.140] By LLM GPT-5.4 0.289 0.411 -0.121 [-0.142, -0.100] Gemma-4-31B-IT 0.314 0.410 -0.095 [-0.100, -0.091] Hermes-4-14B 0.318 0.410 -0.092 [-0.097, -0.087] Llama-3.1-8B 0.337 0.410 -0.072 [-0.078, -0.067] Qwen3-35B-A3B 0.319 0.410 -0.091 [-0.096, -0.086]
4.3 Frontier Alignment: AI Ideas Are Less Aligned with Future Research Frontiers
{tableorg}
[htbp] Next-year field frontier alignment by agent framework and LLM. Frontier alignment is first computed for each broad scientific field and then averaged across fields. The AI and Human columns report the mean frontier coverage of AI-generated ideas and the corresponding follow-on human papers, respectively. AI–Human denotes the difference between AI-generated ideas and follow-on human papers. Brackets report 95% bootstrap confidence intervals. Group AI Human AI–Human [95% CI] Frontier alignment All data 0.285 0.365 -0.080 [-0.085, -0.075] By agent framework Agent Laboratory 0.250 0.377 -0.127 [-0.137, -0.117] AI Scientist v2 0.328 0.360 -0.032 [-0.041, -0.022] Co-Scientist 0.282 0.375 -0.093 [-0.104, -0.081] ResearchAgent 0.234 0.336 -0.102 [-0.111, -0.093] Zero-shot 0.331 0.377 -0.046 [-0.054, -0.038] By LLM GPT-5.4 0.073 0.142 -0.068 [-0.077, -0.060] Gemma-4-31B-IT 0.259 0.391 -0.131 [-0.140, -0.123] Hermes-4-14B 0.302 0.343 -0.041 [-0.049, -0.033] Llama-3.1-8B 0.363 0.391 -0.028 [-0.036, -0.021] Qwen3-35B-A3B 0.269 0.391 -0.122 [-0.132, -0.113]
We next examine the frontier alignment of AI-generated ideas to assess whether they align with future directions of scientific research. As a comparison, we measure the frontier alignment for follow-on human-authored papers that directly cite at least one of the seed papers. To ensure a fair comparison, these follow-on papers are excluded from the next-year corpus when constructing the frontier keyword set. AI-generated ideas and follow-on human papers contain a comparable number of extracted keywords on average (11.80 versus 11.88).
Across research fields, AI-generated ideas cover 28.5% of next-year frontier keywords, compared with 36.5% for follow-on human papers. The difference remains statistically significant across all evaluated research fields. One potential concern is information leakage arising from LLM pretraining on papers published after the seed literature. To mitigate this issue, all AI agents are restricted to retrieving papers available at the time of ideation, preventing explicit access to future publications. Although pretrained LLMs may still implicitly encode knowledge from later papers, AI-generated ideas nevertheless exhibit substantially lower frontier alignment than subsequent human research. Therefore, the reported differences are likely conservative estimates. Together, these results suggest that AI-generated ideas are less aligned with future research directions than the subsequent work produced by human researchers.
4.4 Potential Scientific Impact: AI Ideas are Located in Lower-Impact Regions of the Scientific Landscape
We next examine the potential scientific impact of AI-generated ideas. As a comparison, we also measure the scientific impact of follow-on human-authored papers that directly cite at least one of the seed papers. For human-authored papers, scientific impact is measured using citation counts normalized by publication year and research field. The results show that AI-generated ideas receive lower impact scores than follow-on human papers. Averaged across research areas and study years, the mean potential impact score of AI-generated ideas is 0.387, compared with 0.492 for follow-on human papers, 21.3% lower than that of follow-on human papers (Table 4.4). This pattern holds across 11 of the 12 evaluated scientific fields, with Mathematics being the only field in which the difference is not statistically significant.
Moreover, we validate the neighborhood-based impact proxy using a leave-one-out analysis on human-authored papers. Potential scientific impact scores computed from neighboring papers positively predict the target paper’s own normalized citation score (Spearman , . see SM S2.3 and table S7), supporting the validity of using local neighborhoods to estimate the potential impact of AI-generated ideas.
{tableorg}
[t] Potential scientific impact by agent framework and LLM. Potential scientific impact is first computed for each research area and publication year and then averaged across research areas and years. The AI and Human columns report the mean potential scientific impact scores of AI-generated ideas and the true normalized citations of corresponding follow-on human papers, respectively. AI–Human denotes the difference between AI-generated ideas and follow-on human papers. Brackets report 95% bootstrap confidence intervals. Group AI Human AI–Human [95% CI] Pooled Full Data 0.387 0.492 -0.105 [-0.114, -0.096] By agent framework Agent Laboratory 0.411 0.492 -0.081 [-0.090, -0.073] AI Scientist v2 0.412 0.492 -0.081 [-0.089, -0.072] Co-Scientist 0.424 0.492 -0.068 [-0.076, -0.060] ResearchAgent 0.427 0.492 -0.065 [-0.074, -0.057] Zero-shot 0.423 0.492 -0.069 [-0.078, -0.061] By LLM GPT-5.4 0.472 0.464 0.008 [-0.029, 0.046] Gemma-4-31B-IT 0.441 0.492 -0.051 [-0.059, -0.043] Hermes-4-14B 0.393 0.492 -0.099 [-0.108, -0.091] Llama-3.1-8B 0.383 0.492 -0.109 [-0.118, -0.101] Qwen3-35B-A3B 0.450 0.492 -0.043 [-0.051, -0.035]
4.5 Consistency Across Agent Frameworks and LLMs
We examine whether the four findings remain consistent across different AI agent frameworks and underlying LLMs (Table 4.1–4.4). Overall, the qualitative patterns remain remarkably stable. More sophisticated agent frameworks do not substantially reduce the gaps between AI-generated ideas and human-authored research across the four measures, although modest improvements appear in individual dimensions.
For example, the Zero-shot baseline, the only framework without access to the Semantic Scholar search tool, exhibits the largest exploration-distance gap, indicating that its generated ideas remain most closely anchored to the initial seed papers. Agent frameworks that can retrieve additional literature reduce this exploration-distance gap, suggesting that literature search helps agents move beyond the initial seed context. However, this improvement does not translate into substantially higher frontier alignment or potential scientific impact. Thus, providing access to additional literature alone does not fundamentally alter the overall exploration pattern.
A similar pattern is observed across LLMs. GPT-5.4 is the only evaluated model whose generated ideas occupy semantic neighborhoods with potential scientific impact comparable to subsequent human research. Nevertheless, GPT-5.4 explores a narrower region of the scientific idea space and exhibits substantially lower frontier coverage than human follow-on work.
Taken together, these findings suggest one broader implication. Although all evaluated AI agent frameworks are explicitly instructed to generate novel and high-impact ideas, many explicitly reward novelty through iterative self-reflection and refinement, and all agent frameworks can actively search and retrieve relevant prior literature, current AI research agents do not substantially narrow the gap to human researchers in scientific exploration.
4.6 AI Ideas Primarily Introduce News Methods Rather Than Research Questions
Finally, we examine how AI-generated ideas are constructed. Scientific novelty may arise from identifying new research questions, developing new technical methods, or recombining existing ideas in novel ways [luo2022combination, uzzi2013atypical]. We therefore prompt an LLM to annotate each AI-generated idea into two components: a research question, describing the scientific problem being studied, and one or more methods, describing how the problem is approached. We then compare the extracted research questions and methods against those appearing in the corresponding five seed papers used during ideation (see SM S1.2 and S2.4 for details).
Overall, AI-generated ideas rarely introduce substantially new research questions. Only 10.5% of AI-generated ideas contain research questions that are absent from the seed literature (Figure 4a-b). In contrast, 90.4% introduce new methods that do not appear in the seed papers. These results suggest that when AI-generated ideas differ from prior work, the differences arise predominantly from modifying or recombining methods rather than identifying new scientific problems.
Interestingly, the way AI-generated ideas build on prior work differs across scientific fields (Figure 4c). In engineering and the natural sciences, including Computer Science, Mathematics, Physics, Chemistry, Materials Science, and Engineering, AI-generated ideas almost always retain existing research questions, with most of the apparent difference arising from new methods or methodological recombination. By contrast, Sociology and Business exhibit substantially higher rates of new research questions and correspondingly lower rates of new methods. This pattern is consistent with the nature of these fields, where emerging social and business phenomena frequently motivate new research problems, whereas innovation in engineering and the natural sciences more often takes the form of new methods for addressing established problems.
5 Discussion and Implications
This study suggests a more cautious interpretation of current AI research agents. Recent research introduces increasingly sophisticated agentic mechanisms into scientific ideation, including self-reflection, staged validation, role decomposition, and multi-agent deliberation [lu2026towards, baek2025researchagent, schmidgall2025agent]. These mechanisms can improve the coherence and plausibility of generated research proposals. However, our findings suggest that such capabilities do not necessarily translate into broader scientific exploration. Although these AI agents are specifically asked to propose novel, high-impact, or unlike-prior-work ideas, AI-generated ideas remain substantially more concentrated than human-authored research, stay closer to the starting literature than later human follow-on work, align less with future research frontiers, and are estimated to have less scientific impact.
This distinction matters because scientific discovery is not only about producing plausible ideas, but also about exploring the space of possible ideas. Human scientific progress often involves moving beyond established directions, exploring less familiar regions, and occasionally reframing the underlying research problem itself [foster2015tradition, uzzi2013atypical, fortunato2018science]. From this perspective, current AI research agents appear better suited to local elaboration than exploration. Our findings also suggest that increasingly sophisticated agentic AI frameworks and scaling LLM do not fundamentally resolve this limitation.
More broadly, our findings point toward a future challenge for AI-assisted scientific discovery. The central question may not only be whether AI systems can generate coherent scientific ideas, but whether they can help expand the range of scientific directions. As AI research agents become more deeply integrated into scientific workflows, designing agentic AI systems that broaden scientific exploration will become increasingly important.
References
Supplementary Information
S1 Supplementary Methods
S1.1 Data Sources and Research Area Construction
Overview.
We identify research areas using bibliographic coupling. The main idea is that papers studying related problems tend to cite the same prior work. We therefore represent each paper by the references it cites, transform these high-dimensional reference profiles into compact paper embeddings, and cluster papers with similar embeddings.
Corpus and metadata.
We collect papers and citation links from the Semantic Scholar Academic Graph222https://www.semanticscholar.org/product/api. The raw paper index contains titles, publication years, abstracts, citation counts, reference counts, venues, authors, external identifiers, and Semantic Scholar fields of study. The main ideation analysis uses 12 fields with sufficient title and context coherence after field-level quality checks, including Medicine, Biology, Engineering, Chemistry, Computer Science, Environmental Science, Materials Science, Physics, Mathematics, Economics, Business, and Sociology. We use papers published before 2020 to construct bibliographic-coupling representations. We use papers published between 2020-2025 in our main analysis. Table S1 summarizes the Semantic Scholar corpus used in this study.
| Quantity | Value |
|---|---|
| Semantic Scholar papers retained in paper index | 149,405,218 |
| Papers used for context construction | 75,019,333 |
| Fields used in main ideation analysis | 12 |
| Study years | 2020–2025 |
| Selected research areas | 155 |
| Seed-paper sets | 11,520 |
Bibliographic-coupling for paper representation.
We construct research areas separately within each broad field. For a given field, let denote its set of papers and the set of corpus papers they cite.
We represent each paper by its references, following the logic of bibliographic coupling [kessler1963bibliographic]: papers that cite overlapping prior literature tend to address related problems. Under a matrix formulation, if denotes the paper–reference matrix, then the standard bibliographic-coupling matrix is given by
where each entry counts the number of references jointly cited by two papers. The raw coupling matrix captures shared citation structure, but it treats all references equally. In practice, some highly cited papers (e.g., widely used optimization methods or foundation architectures) are cited across many unrelated research areas and therefore provide relatively weak evidence of topical similarity.
Therefore, for each field, we construct paper citation embeddings directly from the paper–reference matrix . To reduce the influence of broadly cited references, we weight each reference column by an inverse-document-frequency term, so that references cited by many papers receive lower weight and more field-specific references receive higher weight. The weighted rows are L2-normalized, projected to dimensions using truncated SVD, and L2-normalized again to produce the bibliographic-coupling embeddings used for subsequent research area identification.
Research area identification.
Research areas are identified by clustering the papers published in 2020–2025. Within each field, MiniBatchKMeans [sculley2010web] is applied to the embeddings to obtain candidate citation-defined areas, and papers are assigned to their nearest cluster centroid. We use MiniBatchKMeans because the corpus contains millions of papers per field, requiring a scalable clustering method that can be applied consistently across all analyzed fields. We retain only active areas that contain papers in every study year from 2020 through 2025. This longitudinal filter yields 11,520 seed-paper sets that cover 155 distinct research areas across the 12 analyzed fields. Table S2 summarizes the number and examples of selected research areas in each field.
| Field | Areas | Representative selected areas |
|---|---|---|
| Biology | 10 | Arabidopsis genetics; stem-cell differentiation; oocyte maturation; stem-cell culture; evolutionary cooperation; microbial community assembly |
| Business | 19 | Supply chains; insurance claims; supplier selection; digital platforms; live streaming; COVID-19 business disruption; financial reporting |
| Chemistry | 13 | DNA origami; gold nanoparticles; electronic structure; ion-mobility lipidomics; lipid membranes; molecular spectroscopy; polymer materials |
| Computer Science | 9 | Cognitive radio networks; error-correcting codes; image fusion; educational AI systems; machine-learning applications; networked agents; statistical evaluation |
| Economics | 18 | Synthetic control; time-series forecasting; propensity-score methods; digital currency; instrumental variables; treatment effects; financial risk |
| Engineering | 12 | Road traffic systems; ant-colony optimization; satellite communications; optogenetic control; power-system stability; structural monitoring; wireless networks |
| Environmental Science | 11 | Bayesian ecological modeling; parasite communities; persistent pollutants; heavy-metal contamination; wastewater treatment; microbial ecology; forest governance |
| Materials Science | 14 | Monolayer graphene; gold nanoparticles; first-principles materials modeling; lithium-ion batteries; iron oxide; nanocomposites; thin films |
| Mathematics | 10 | Partially hyperbolic dynamics; mapping class groups; least-squares estimation; Banach spaces; simple groups; partial differential equations; stochastic processes |
| Medicine | 9 | Metabolic syndrome; physical activity; pancreatic cancer; infectious disease; parasites; body-mass index; biomarkers; clinical risk factors |
| Physics | 15 | Quantum simulation; nonlinear wave equations; atomic gravimetry; solar coronal mass ejections; photonic crystals; finite-element methods; heavy-ion collisions |
| Sociology | 15 | Socioeconomic status; family planning; collective efficacy; aging; older adults; Indigenous communities; social inequality; public health behavior |
| Total | 155 |
S1.2 Generating Scientific Ideas with AI Agents
This appendix provides additional implementation details for the ideation experiments described in the main text.
Seed-paper contexts.
We sample 11,520 seed-paper sets across the research areas, drawing 8 research areas per field per year and 20 anchor papers per area across the 2020–2025 study years. Each set contains five papers: one anchor paper and four related papers from the same research area. Related papers are selected from papers no later than the anchor year. All agent frameworks receive the same five-paper context for a given run, and the context contains only paper titles and abstracts.
AI Research Agent framework.
We evaluate five AI research agent frameworks. The Zero-shot agent receives the five-paper context once and directly generates one research idea. AIScientist receives the same context, generates an initial idea, and then refines it through self-reflection rounds. In this study, we evaluate only this ideation stage of AIScientist [lu2026towards], and we do not run its later experiment-execution tree-search stage. ResearchAgent [baek2025researchagent] uses the same five-paper context to propose a research problem, develop a method, and design experiments, with validator agents scoring intermediate outputs before final selection. AgentLaboratory [schmidgall2025agent] receives the same literature context as a short literature review and uses a dialogue between role-based agents, including a postdoc agent and a PhD-student agent, to formulate a final research plan. Co-Scientist [gottweis2026coscientist] is implemented based on the published description of the framework: multiple agents generate, reflect on, rank, evolve, and synthesize competing hypotheses before a final meta-review.
Prompts and tools.
This section documents the prompts and tools used to generate ideas. Each system receives the same five paper titles and abstracts from the paper corpus available before time , but the systems differ in how they turn that evidence into a proposal. Zero-shot prompting uses one direct JSON prompt. AIScientist uses iterative ideation and reflection with a literature-search command. ResearchAgent uses separate problem, method, and experiment stages. AgentLaboratory uses a dialogue between role-based agents to produce a final plan. Co-Scientist uses supervisor parsing, literature search, hypothesis generation, reflection, tournament ranking, evolution, and meta-review stages.
The placeholder {five_paper_literature_context} in the prompt listings denotes the rendered five-paper context used at runtime. It is a numbered list containing only the title and abstract of each seed paper. For the zero-shot baseline, this rendered block is inserted directly at the beginning of the user message. For AIScientist, the same rendered block is passed as {workshop_description} in the initial idea-generation prompt. For ResearchAgent, the first seed paper is passed through the original implementation’s main-paper field, the remaining four seed papers are passed as references, and the entity list is left empty; our wrapper also keeps the full five-paper list for the simplified fallback prompts. For AgentLaboratory, the five papers are converted into the condensed literature-review field shown to the postdoc and PhD-student agents. For Co-Scientist, the five papers are passed to the supervisor stage as the initial research goal and literature context. Later stages use the parsed research problem, retrieved literature, and intermediate hypotheses. Other braced fields in the listings are filled with intermediate outputs from earlier stages of the same agent run.
Execution protocol.
Each generation run starts from one seed-paper set, one agent framework, and one LLM. The zero-shot baseline makes a single model call and returns the JSON idea from that call. AIScientist first receives the five-paper context in the idea-generation prompt, then runs for five ideation/reflection rounds; at each round the model either issues a SearchSemanticScholar action or returns a FinalizeIdea action. The search action is served by a local literature-search wrapper over papers available before time , so the agent never observes follow-on papers from the evaluation period.
ResearchAgent runs three stages in order: problem identification, method development, and experiment design. In the full ResearchAgent path, each stage is generated and validated for two iterations, and the highest-scoring candidate according to the validator is passed to the next stage. AgentLaboratory starts with the postdoc agent, alternates between postdoc and PhD-student turns for up to eight plan-formulation steps, and treats the postdoc’s PLAN command as the final idea. Co-Scientist first generates competing hypotheses from the shared literature context, reflects on each hypothesis, ranks the candidates, and then writes a final synthesized proposal through a meta-review stage.
If an agent does not produce the required final structured output, the run is marked invalid and excluded by the validity filter below.
Prompt listings.
The boxes reproduce the prompt text used in the generation runs. For systems with multiple stages, the boxes are ordered in the same sequence as the generation process.
Zero-shot baseline.
AIScientist.
We use the original AIScientist ideation setup, which consists of a system prompt, an idea-generation user prompt, and a reflection user prompt. The agent runs for five rounds of ideation and reflection. It may call
SearchSemanticScholar
before finalizing an idea with
FinalizeIdea
. The prompt names this literature-search command as
SearchSemanticScholar
; in our runs, the command returns papers from our local paper corpus restricted to papers available before time .
AIScientist system prompt.
AIScientist idea-generation user prompt.
AIScientist reflection user prompt.
ResearchAgent.
ResearchAgent formulates an idea through three stages: problem identification, method development, and experiment design. Each stage uses a role-specific system prompt and a user prompt. The evaluated ResearchAgent idea is constructed from the generated problem, method, and experiment plan.
ProblemIdentifier system prompt.
ProblemIdentifier user prompt.
MethodDeveloper system prompt.
MethodDeveloper user prompt.
ExperimentDesigner system prompt.
ExperimentDesigner user prompt.
Validator system prompts.
AgentLaboratory.
AgentLaboratory uses a dialogue between a Postdoc agent and a PhD-student agent during plan formulation. We evaluate the final
PLAN
command produced in this phase. In our setting, this plan-formulation phase uses the provided literature context and dialogue history.
Shared inference templates.
Postdoc role description and phase prompt.
Postdoc command descriptions.
PhD-student role description and phase prompt.
PhD-student command descriptions.
Co-Scientist.
Co-Scientist follows the published generate–debate–evolve workflow. A supervisor agent parses the research goal and search queries; generation agents propose hypotheses; reflection agents critique them; ranking agents compare hypotheses by novelty, significance, feasibility, and testability; evolution agents revise top hypotheses; and a meta-review agent synthesizes the final proposal.
Co-Scientist supervisor prompt.
Co-Scientist generation prompt.
Co-Scientist reflection prompt.
Co-Scientist ranking prompt.
Co-Scientist evolution prompt.
Co-Scientist meta-review prompt.
For reproducibility, we store the rendered prompt text, raw model responses, parsed actions, and intermediate stage outputs for each run.
Output standardization.
The evaluated AI research-agent frameworks produce outputs in substantially different formats, including structured JSON ideas, staged research proposals, markdown plans, and multi-agent dialogue traces. Before analysis, we first convert the output of each framework into a standardized generated-idea document. For Zero-shot, AIScientist, and Co-Scientist, we use the generated title (or name), hypothesis, and abstract-like proposal text. For ResearchAgent, we use the proposed research problem and method. For Agent Laboratory, we use the main proposal sections of the final research plan. This standardization produces a unified textual representation of every generated idea.
Although the generated ideas have been standardized into a common textual format, they remain heterogeneous with respect to human-authored papers, which are represented by titles and abstracts. To enable a unified comparison, we further convert both AI-generated ideas and human-authored papers into a common scholarly annotation schema following recent scientific-article key-insight extraction approaches [song2025scientific]. Specifically, we prompt Gemma-4-31B-IT to extract the Aim, Motivation, Research Question, Technical Method, and Scholarly Keywords for every document. The extracted research questions and technical methods are concatenated and encoded into the shared semantic embedding space used throughout the paper.
The extraction prompt is shown below:
Idea Generation and Validity Filtering.
For the 11,520 seed-paper sets, we evaluate four open-source LLMs under five AI agent frameworks, yielding 230,400 generation runs. Due to budget constraints, experiments using the proprietary GPT-5.4 model are conducted only on randomly sampled seed-paper sets from 2022, resulting in an additional 2,400 generation runs. In total, the study includes 232,800 AI idea-generation runs.
We next apply validity filtering to remove unsuccessful generations. A generation run is considered valid if the agent completes successfully and produces a non-empty structured output from which a research idea can be extracted. Runs that fail to complete, return unparsable structured outputs, or produce an empty final proposal are excluded. This filtering yields 219,655 valid AI-generated ideas from 232,800 generation runs, corresponding to a validity rate of 94.4%.
Table S3 summarizes the numbers of generation runs and valid ideas by publication year, AI agent framework, and LLM. Table S4 presents representative AI-generated ideas produced by Gemma-4-31B-IT under different agent frameworks using the same seed-paper set.
| Dimension | Categories | Generation runs |
|---|---|---|
| Overall | ||
| Total open-weight runs | 230,400 | |
| GPT-5.4 subset runs | 2022 representative subset | 2,400 |
| Total generation runs | Open-weight + GPT-5.4 | 232,800 |
| Total AI-generated Ideas | Valid output | 219,655 |
| By year (1,920 seed sets 5 agents 4 LLMs) | ||
| 2020 | 1,920 seed sets | 38,400 |
| 2021 | 1,920 seed sets | 38,400 |
| 2022 | 1,920 seed sets | 38,400 |
| 2023 | 1,920 seed sets | 38,400 |
| 2024 | 1,920 seed sets | 38,400 |
| 2025 | 1,920 seed sets | 38,400 |
| By agent (11,520 seed sets 4 LLMs) | ||
| Agent Laboratory | 1 agent | 46,080 |
| AIScientist | 1 agent | 46,080 |
| ResearchAgent | 1 agent | 46,080 |
| Co-Scientist | 1 agent | 46,080 |
| Zero-shot | 1 agent | 46,080 |
| By LLM (11,520 seed sets 5 agents) | ||
| Gemma-4-31B-IT | Open-weight | 57,600 |
| Llama-3.1-8B | Open-weight | 57,600 |
| Hermes-4-14B | Open-weight | 57,600 |
| Qwen3-35B-A3B | Open-weight | 57,600 |
| GPT-5.4 | Proprietary | 2,400 |
| Agent | Generated idea |
|---|---|
| 2025 run 1: long-range graph representation learning.Learning Long Range Dependencies on Graphs Via Random Walks [chen2025learning]; Polynormer [deng2024polynormer]; Classic GNNs Are Strong Baselines [luo2024classic]; Diffusing to the Top [lin2025diffusing]; Predicting Global Label Relationship Matrix under Heterophily [liang2023predicting]. | |
| Zero-shot | PolyWalker. Replaces random-walk sequence encoders with polynomial-expressive linear attention to capture long-range graph dependencies efficiently. |
| AIScientist | GraphMamba. Uses selective state-space models over random-walk sequences to scale long-range graph dependency modeling with linear memory. |
| ResearchAgent | PolyWalk-LR. Combines local message passing, random-walk embeddings, polynomial attention, and a low-rank global label-relation matrix. |
| Agent Lab. | PolyDiff-GNN. Adds a plug-and-play polynomial diffusion layer to classic GNNs to model multi-hop dependencies without graph-transformer cost. |
| Co-Scientist | AdaptiveWalk-GNN. Learns node-adaptive gates over random-walk sequence features, polynomial diffusion scales, and local message passing, with label-relation regularization for heterophilous graphs. |
| 2025 run 2: causal visual reasoning and tool-use agents.NarrativeBridge [nadeem2025narrativebridge]; CS-Bench [song2025csbench]; TACT [caciularu2024tact]; VisMin [awal2024vismin]; GTA: A Benchmark for General Tool Agents [wang2024gta]. | |
| Zero-shot | CausalVisTool. Combines causal-temporal video narratives with tool-use evaluation to test whether multimodal agents preserve causal consistency. |
| AIScientist | Counterfactual-VideoBench. Turns causal video understanding into a counterfactual intervention benchmark for testing whether VLMs rely on temporal correlations. |
| ResearchAgent | MATR. Links fine-grained visual perception, aggregative information extraction, and tool execution in a perceive–aggregate–execute pipeline. |
| Agent Lab. | CCTR. Evaluates causal tool reasoning by asking agents to connect dynamic visual events with the correct tool actions and counterfactual variants. |
| Co-Scientist | InterveneBench. Pairs minimally edited counterfactual videos with executable tool trajectories to test whether agents revise tool choices when causal events change while irrelevant visual content remains fixed. |
S1.3 Definitions of Exploration Measures
This section formalizes the four measures introduced in the main text. Each AI-generated idea or human-authored paper is represented by its standardized text, from which we compute a text embedding and extract scholarly keywords. Below, denotes the L2-normalized embedding of an idea or paper . In the paper, we use Qwen3-Embedding-4B [zhang2025qwen3] as the text embedding model.
Exploration breadth measures how widely a set of ideas or papers spreads within the same research area. Let contain either the AI-generated ideas from research area or human-authored papers from that area. Breadth is the mean pairwise cosine distance,
The values reported in the main text average this quantity over research areas within each comparison group (pooled, by agent framework, or by LLM).
Exploration distance measures how far an idea or paper moves from the seed literature. Each generation run starts from a set of five seed papers, summarized by the normalized centroid
AI-generated ideas from run and follow-on human papers citing at least one paper in are scored by their cosine distance to this centroid,
so that ideas and papers stay close to the seed literature receive small distances.
Frontier alignment measures how well a group of ideas or papers covers the topics that become prominent in the following year. For field and seed year , the next-year frontier is the set of the top 10% most frequent scholarly keywords among human-authored papers published in field in year , excluding the evaluated follow-on papers. For each comparison group , the keywords extracted from its ideas or papers are pooled into a set , and frontier alignment is the share of the frontier covered by this pool,
Potential scientific impact measures whether an idea or paper falls near historically influential parts of its research area. Each human-authored paper from research area and publication year receives a normalized citation score
where is its citation count and is the leave-one-out mean of over the other papers from the same area and year. The potential scientific impact of an AI-generated idea or follow-on human paper is the mean score of its nearest human-authored papers in the same area,
where contains the nearest papers by cosine similarity, restricted to papers published no later than the seed year.
S2 Supplementary Discussion
S2.1 Results by Scientific Field
The main text reports average results across the 12 broad scientific fields. Here we present field-level results for all four measures. For each measure, the reported gap is defined as the AI-generated idea score minus the corresponding human-paper score. Negative values therefore indicate that AI-generated ideas have lower exploration breadth, shorter exploration distance, lower frontier alignment, or lower potential scientific impact than the corresponding human-authored papers. The detailed results are presented in Table S5.
| Field | Breadth gap | Distance gap | Frontier gap | Impact gap |
|---|---|---|---|---|
| Computer Science | -0.032∗∗ | -0.069∗∗ | -0.068∗∗ | -0.161∗∗ |
| Business | -0.041∗∗ | -0.059∗∗ | -0.084∗∗ | -0.143∗∗ |
| Sociology | -0.045∗∗ | -0.037∗∗ | -0.119∗∗ | -0.109∗∗ |
| Materials Science | -0.024∗ | -0.086∗∗ | -0.103∗∗ | -0.101∗∗ |
| Engineering | -0.047∗∗ | -0.079∗∗ | -0.066∗∗ | -0.089∗∗ |
| Chemistry | -0.027∗∗ | -0.072∗∗ | -0.066∗∗ | -0.065∗∗ |
| Environmental Science | -0.030∗ | -0.052∗∗ | -0.071∗∗ | -0.058∗∗ |
| Medicine | -0.025∗ | -0.040∗∗ | -0.141∗∗ | -0.050∗∗ |
| Economics | -0.041∗∗ | -0.083∗∗ | -0.039∗∗ | -0.037∗ |
| Physics | -0.023∗∗ | -0.056∗∗ | -0.050∗∗ | -0.032∗ |
| Biology | -0.028∗∗ | -0.088∗∗ | -0.052∗∗ | -0.028∗ |
| Mathematics | 0.008 | -0.045∗∗ | -0.101∗∗ | -0.006 |
S2.2 Robustness Check: Centroid-Based Measure of Exploration Breadth
We measure exploration breadth using an alternative centroid-based approach. For each research area and each comparison group (i.e., AI-generated ideas produced by a specific agent framework, LLM, and publication year), we compute the normalized centroid of the semantic embeddings and measure the cosine distance between each AI-generated idea and its corresponding centroid. Larger distances indicate greater exploration breadth, whereas smaller distances indicate that papers or ideas are more tightly concentrated around the centroid of their research area.
Table S6 reports the resulting centroid-distance statistics across AI agent frameworks and LLMs. Consistent with the main pairwise-similarity analysis, AI-generated ideas remain closer to their area centroids than human-authored papers do. The pattern holds for each agent framework and each LLM, including Co-Scientist and GPT-5.4.
| Group | Mean dist. | Median dist. |
|---|---|---|
| AI ideas | 0.340 | 0.335 |
| Human papers | 0.362 | 0.355 |
| AI ideas by agent framework | ||
| Zero-shot | 0.319 | 0.314 |
| AIScientist | 0.320 | 0.315 |
| ResearchAgent | 0.338 | 0.329 |
| Agent Lab. | 0.319 | 0.315 |
| Co-Scientist | 0.334 | 0.326 |
| AI ideas by LLM | ||
| Llama-3.1-8B | 0.319 | 0.314 |
| Hermes-4-14B | 0.325 | 0.318 |
| Gemma-4-31B-IT | 0.332 | 0.328 |
| Qwen3-35B-A3B | 0.340 | 0.335 |
| GPT-5.4 | 0.303 | 0.298 |
S2.3 Validation of the Potential Scientific Impact Measure
We estimate the potential scientific impact of AI-generated ideas using the average normalized citation score of their 20 nearest human-paper neighbors. Here we validate this neighborhood-based impact measure using a leave-one-out analysis on human-authored papers. Specifically, for each target human-authored paper, we remove the target paper itself and compute the average normalized citation score of its 20 nearest prior human-paper neighbors from the same research area. We then examine whether this neighborhood-based impact score predicts the target paper’s own normalized citation score.
Table S7 shows that the neighborhood-based impact score is positively associated with the target paper’s subsequent citation performance (Spearman , Pearson ; both ). Thus, papers located in semantic neighborhoods with historically higher citation impact are themselves more likely to become highly cited. Although local semantic neighborhoods explain only part of the variation in citation outcomes, these results support the use of neighborhood citation statistics as a proxy for estimating the potential scientific impact of AI-generated ideas.
| Measure | Value |
|---|---|
| Human papers evaluated | 16,500 |
| Mean prior neighbors | 19.9 |
| Spearman correlation | 0.155∗∗ |
| Pearson correlation | 0.166∗∗ |
S2.4 Validation of Research Question and Method Annotation
The analysis in Section 4.6 relies on identifying the research question and technical methods of each AI-generated idea and determining whether they are already present in the corresponding five seed papers. To evaluate the reliability of this annotation procedure, we conduct an independent LLM-based validation following recent work on LLM annotation reliability [gilardi2023chatgpt].
Specifically, three independent LLM annotators, including Qwen-30b, Llama-8B and Gemma-31B, are given the same AI-generated idea together with its corresponding five seed papers. Each annotator independently determines whether the generated idea introduces (i) a research question that is absent from the seed literature and (ii) a technical method that is absent from the seed literature. Table S8 summarizes the agreement among the three annotators. Agreement is consistently high for both research question and research method judgments. All three annotators agree on 74.0% of research-question labels and 77.6% of technical-method labels. Pairwise agreement ranges from 80.8% to 89.9%, while Gwet’s AC1 exceeds 0.77 for both tasks despite the class imbalance. These results indicate that determining whether an AI-generated idea introduces a new research question or a new method relative to the seed literature is a reliable annotation task.
| Agreement metric | Research question | Method |
|---|---|---|
| All three annotators agree | 74.0% | 77.6% |
| Gwet’s AC1 | 0.771 | 0.809 |
| Qwen 30B vs. Llama 8B | 82.7% | 80.8% |
| Qwen 30B vs. Gemma 31B | 82.9% | 84.5% |
| Llama 8B vs. Gemma 31B | 82.3% | 89.9% |