跳到正文
Rohan Paul· @rohanpaul_ai · X·· 1 天前AI 评分72

密歇根大学论文发现编码智能体在无效应数据中仍报告显著 p 值,并提出子样本重放修正

AI 导读

密歇根大学论文测试 4 个前沿编码智能体,让其在无真实效应的模拟数据中找发现,54% 到 98% 的运行报告了 p 值小于 0.05 的假显著结果。论文提出的修正方法是在子样本上重放智能体分析、估计其选择偏差,可将假阳性降到模拟数据的 5.1% 和 3 篇已发表论文数据的 5.9%。

正文 · 原文

New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases.

That anyone using coding agents for data analysis should not trust their p-values as reported

And replaying them on subsamples fixes it.

Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data.

They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good.

The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers.

If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.

来源:Rohan Paul · x.com