递归自我改进（RSI）这一概念可以追溯到I. J. Good（1965），他将“超智能机器”定义为一个能够在所有智力活动中超越人类、并设计出更好的机器来改进自身的系统。Yudkowsky（2008）用“递归自我改进”一词来指代一个特定的反馈回路：AI 利用其当前的智能来改进产生其智能的认知机制。
在现代 AI 中，这一反馈回路可能意味着模型直接重写自身的权重，或者更广义地说，模型改进训练流程和部署系统，进而催生出性能更优的后继模型，在经济上有价值的任务中表现全面提升。前沿实验室的研究开发速度已被证明会急剧加快（Anthropic；OpenAI）。
我特意提到“部署系统”，是因为原始模型与真实世界上下文之间的这一层，似乎与模型的原始智能（即预训练后立即进行的评测所衡量的能力）同等重要。Harness 是 AI 部署的重要组成部分，Claude Code 和 Codex 等成功的编程智能体产品已经证明了这一点。Harness是围绕基础模型的系统，它编排执行过程，决定模型如何思考和规划、如何调用工具并采取行动、如何感知和管理上下文、如何存储产物，以及如何评估结果。
这篇文章将聚焦于围绕 harness 工程的研究，以及它如何推动 RSI。近期许多关于自动研究、自我改进智能体和进化式程序搜索的工作，都可以围绕这一问题来组织。其他关于模型自我对弈、合成数据、测试时训练以及更广泛的持续学习主题的工作，也符合 RSI 的愿景（例如 Yuan et al. 2024、Chen et al. 2024）、Zhao et al. 2025、Choi et al. 2026），但它们不会是本文的重点。
Harness 设计模式
与早期智能体框架相比，“智能体 = LLM + 记忆 + 工具 + 规划 + 行动”，而 harness 工程还额外包括工作流设计（例如循环工程）、评估、权限控制和持久状态管理。它不再只是提示词模板，而更接近运行时与软件系统设计：模型如何观察、行动、记忆、自我检查并改进。
设计应当刻意保持简单和通用，以实现泛化，很可能需要参考现有的软件工程实践，以从预训练知识中获益。操作系统与 harness 之间也存在很强的类比关系。与操作系统类似，harness 应当封装复杂的逻辑，同时保持接口简单。与此同时，配置、工具接口和其他协议可能会逐渐在整个行业范围内实现标准化。
模式 1：工作流自动化
定义一个模型能够操作、测试并迭代的工作流，是自动化的一项关键设计。Karpathy 的 autoresearch 仓库（https://github.com/karpathy/autoresearch）就是一个清晰示例，展示了如何构建这样的工作流。一种常见的工作流遵循目标导向的循环：规划、执行、观察/测试、改进，然后再次执行，直到目标达成。该过程可能会主动向用户发起请求，以澄清任务规格或执行偏好。
（图片来源：OpenAI codex agent post）
该工作流图还强调，模型要分析自身的轨迹和失败案例，然后通过“智能体运行时”而非静态提示词模板来迭代推进。
模式 2：文件系统作为持久记忆
长时程智能体系统中反复出现的一种模式，是对丰富状态和产物进行简单控制。一个 harness 不应把整个工作流和所有日志都放在上下文中；相反，它应把持久状态保存在文件里。在长时程智能体 rollout 中，实验日志、代码 diff、论文摘要、错误追踪和过往 rollout 轨迹等产物，往往会增长到远超模型训练时所支持的上下文窗口。
学习如何读取、写入和编辑文件系统（通常通过 bash 命令）是大语言模型的一项基础技能，因此以文件这种简单形式管理持久记忆，自然也能从核心模型能力的提升中获益。
模式 3：子智能体与后台任务
一个 harness 可以派生多个子智能体并行执行，并监控后台任务。当主智能体需要搜索多个假设、并发运行实验，或在不污染主上下文的情况下委派相互隔离的子任务时，这一点非常有用。随后，父智能体需要一个小型进程管理器：启动任务、检查日志、取消失败的运行，并将结果合并回主智能体线程。
关键的设计选择是让并行性显式且可检查。如果子智能体的输出只存在于短暂的对话上下文中，它们很快就会过时并被隐藏。如果它们以文件、日志和状态记录的形式存储，模型就能在中断后恢复，并对自身的执行历史进行推理。
案例研究：编程智能体 harness
主流编程智能体的核心接口在 Claude Code、Codex、OpenCode 以及 Cursor 风格的智能体之间已经趋于稳定。它们通常使用类似这样的循环：
借助一组工具，编程智能体能够在给定代码仓库中开发和调试问题，就像人类开发者配备 IDE 一样。
（并非完整列表；仅为演示而展示。如有兴趣，请阅读这篇。）
| 分组 | 工具定义 |
|---|---|
| 文件系统 | - 文件发现：glob、grep、ls- 文件读取： read、read_many- 文件修改： write（一个全新文件）；edit（字符串精确匹配替换）；multi_edit；apply_patch（应用结构化补丁/diff） |
| Shell 执行 | 运行命令：bash、PowerShell |
| IO | lsp、git 工具如 git_status、git_diff、git_commit |
| 外部上下文 | MCP 工具、技能 |
| 网页搜索 | web_search、web_fetch、浏览器工具 |
| 产物 | 读取文档、图像；生成 HTML、图像 |
| 后端处理 | 例如：CronCreate、CronDelete、CronList |
| 智能体委派 | 例如：spawn_agent、resume_agent、wait_agent、list_agents、close_agent、interrupt_agent 等。 |
框架层 vs 核心智能？
很难预测 RSI 的未来在多大程度上会依赖框架工程，但 RSI 的近期路径不太可能一开始就是模型直接重写自己的权重。我对一条切实可行的近期路径的预测是：
- 框架工程将朝着元方法论的方向演进（即改进获取更好答案的机制，而不仅仅是改进答案本身）。框架系统本身成为优化目标，启发式规则更少，通用机制更多。
- 反过来，成熟的框架能够实现自动研究，形成模型自我改进的闭环，而更智能的模型则防止框架过度工程化，保持系统的可持续性。
最终，许多 harness 改进有可能被内化到核心模型行为中，但与外部上下文和工具的接口应当保留。我们在提示词工程中已经看到了这一模式的较温和版本：随着指令微调和模型推理能力的提升，手动提示词技巧变得不再那么核心，但指定目标、约束、上下文和评估的需求并未消失。
Harness 优化
在 harness 系统中，被优化的对象大致经历了这样的演进：指令提示词 → 结构化上下文 → 工作流 → harness 代码 → 优化器代码。随着模型变得越来越智能和强大，我们朝着更复杂的目标和更通用的方法迈进。
上下文工程
简单地将所有工具响应和模型生成内容追加到上下文中，随着智能体任务时间跨度的显著增加，很快就会失控。上下文管理是一个为 LLM 构建更结构化、更简洁的上下文并管理持久状态的层。毫无疑问，长上下文研究将持续取得进展，但目前长上下文智能与上下文工程有时交织在一起。
Agentic Context Engineering（ACE；Zhang et al. 2025）将上下文视为一份不断演进的行动手册，而非一份不断变长的提示词。它包含三个组件，用于维护一份由要点条目组成的上下文行动手册，每个条目都有一个标识符和一段描述。
- 生成器：参考要点条目，生成任务轨迹。
- 反思器：从成功和失败的轨迹中提炼洞见。
- 策展器：以增量式、逐条列出的条目更新结构化上下文。
为防止迭代重写过程中出现上下文坍缩和简洁性偏差，ACE 的一个关键设计选择是：策展器不重写整个提示词块。它改为输出一组结构化的、逐条列出的要点条目，形式为（标识符，描述），这些条目通过确定性逻辑合并到一份结构化上下文日志中。上下文条目会定期进行精炼和去重。
ACE 能够从 rollout 中学习洞见，这帮助我们朝着自我管理的记忆迈进，但更新规则和整体工作流仍然是手工设计的。为了迈向更具自我改进能力的循环，Meta Context Engineering（MCE；Ye et al. 2026）将机制（如何管理上下文）与产物内容（上下文中包含什么）分离，在元优化层面运行技能演化，在基础层面运行上下文优化。
一个 MCE 技能 $s \in \mathcal{S}$ 定义了一个上下文函数 $c_s=(\rho_s,F_s)$，并将输入 $x$ 映射到上下文 $c = F_s(x;\rho_s)$，其中：
- $\rho_s = \{\rho_1,\dots,\rho_m\}$ 是静态组件（提示词、知识库、代码库）。
- $F_s = \{F_1,\dots,F_k\}$ 是动态算子（搜索、选择、过滤、格式化）。
双层优化是在训练数据上给定技能 $s$ 的情况下找到最佳上下文 $c_s^*$，而外层循环则找到在验证集上提供最佳性能的最优技能：
$$ \text{Inner: }c_s^*=\arg\max_{c_s}J_\text{train}(c_s;s)\quad \text{Outer: }s^*=\arg\max_{s\in\mathcal{S}}J_\text{val}(c_s^*) $$
技能数据库会记录此前技能、上下文函数与评估指标的历史 $\mathcal{H}_{k-1} = \{(s_i,c_i,J_i^\text{train}, J_i^\text{val})\}_{i=1}^{k-1}$。一个元层级智能体会在给定任务 $\tau$ 的情况下，对先前的技能执行智能体式的 交叉以创建新技能：$s_k=\text{crossover}(\tau,\mathcal{H}_{k-1})$。
随后，一个基础层级的上下文工程师执行技能 $s_k$，并在当前技能的引导下，从 rollout 反馈 $\mathcal{R}_k$ 中学习上下文函数：$c_k=\text{engineer}(\tau,s_k;c_{k-1}^*,\mathcal{R}_k)$。
MCE 不像 ACE 那样强制采用某种启发式规则来组织上下文。它使用 自由形式的技能来存储任务中最重要的知识，并迭代地共同演化技能与以技能为条件的上下文。在实现层面，上下文函数 $c$ 被实例化为一个专用目录中的文件集合，既包含静态组件（skill.md），也包含动态组件（上下文与数据 rollout）。元层级和基础层级的优化都在具备标准工具集的智能体式编码环境中执行，
$$ \mathcal{T}=\{\texttt{Read},\texttt{Write},\texttt{Edit},\texttt{Bash},\texttt{Glob},\texttt{Grep},\texttt{TodoWrite}\} $$
Meta-Harness（Lee et al. 2026）又深入了一层：被优化的对象是代码，这些代码决定并优化应当存储、检索以及呈现给模型的信息。其名称中的“Meta-”意味着它是一个用于优化 harness 的 harness。
用于创建新 harness 的提议者本身就是一个编码智能体，最终输出是位于帕累托前沿上的一组 harness 候选。
- 整个执行历史可通过文件系统访问，因此编码智能体使用诸如
grep或cat之类的命令来读取它，而不是把所有内容都塞进单个提示词上下文中。 - 所提议的 harness 是文件系统中的一个字典，包含其自身的源代码、分数、rollout 轨迹以及状态更新。
- mete-harness 循环迭代地创建新的 harness，只有合格的才会被保留。
不过，重要的教训很明确：一旦 harness 设计变成可执行的搜索空间，强大的编程智能体就能利用与人类工程师相同的设计空间。
工作流设计
harness 工程中的工作流设计可以由领域专家手工打造。以自动研究为例，已有多种框架被提出并经过测试。AI Scientist 系统（Lu et al. 2026）构建了一条流水线，用于提出研究想法、编写代码、运行实验、分析结果、撰写论文稿件并进行同行评审。Meng et al. (2026) 将可验证性作为 ScientistOne 的核心设计约束，其中每一项主张（引用、数值、方法、结论）都必须追溯到证据来源，并通过 Chain-of-Evidence 检查进行审计。
Autodata 智能体（Kulikov et al. 2026）被设计为像数据科学家一样工作，用于生成训练和评估数据。主智能体管理一个提出问题的 challenger、一个 weak solver、一个 strong solver 以及一个 verifier/judge，目标是在“恰到好处”的难度水平上合成数据，即 strong solver 成功而 weak solver 失败。
在 Autodata 中，挑战者提示词会根据求解器和验证器的反馈进行迭代更新。这里的局限在于，合成任务被用于微调弱求解器，而非强求解器；如果这个循环无法迭代地改进强模型，那它更像是在一个生成的提示词分布上进行间接蒸馏，RSI 的意味更弱。
工作流的设计空间极其庞大，我们自然可以把工作流设计视为一个搜索问题，因此应当能够通过算法找到好的解决方案，而不只是靠人工手工打造。沿着这个方向，智能体系统的自动化设计（ADAS；Hu et al. 2025）将智能体设计本身形式化为一个优化问题，即“元智能体搜索”，由一个元智能体提出新的智能体工作流设计。
- 用一个智能体工作流档案库进行初始化，其中包含 CoT、self-refine 等简单智能体。
- Ask a meta-agent to program new agents, all in code, inspired by existing solutions in the archive.
- 元智能体首先生成新工作流的高层描述，然后用代码将其实现。
- 随后，这份草拟程序会经过元智能体的两个 self-refine 步骤（即先让模型提供反馈，再让同一个模型根据反馈改进先前生成的输出；Madaan et al. 2023），以检查其新颖性。
- 评估每一个新的候选方案，并将成功的方案加回存档。
- 重复步骤 2-3，直到达到最大迭代次数。
（图片来源：Hu et al. 2025）
AFlow（Zhang et al. 2025）将智能体工作流表示为一个图，其中节点代表调用 LLM 的动作，边则以代码实现逻辑运算。工作流优化依赖于 MCTS（蒙特卡洛树搜索）：
- 用模板在树中初始化起始工作流 $W_0$。
- 使用分数与均匀探索的软混合策略选择一个工作流节点。
- 通过要求 LLM 以其评估表现为条件生成一个修改后的工作流来扩展该节点。
- 执行并评估新的工作流。
- 如果新工作流在 $N$ 轮的预算内表现出改进，则将其加回树中。
- 重复步骤 2-5，当 top-$k$ 平均分数趋于平稳或达到预算时停止。
AFlow 在问答、代码和数学任务中的实验表明，相比人工设计的工作流和 ADAS，AFlow 取得了不错的提升。
自我改进的 Harness
无论是上下文工程还是工作流设计，都只是 harness 的一部分。我们需要搜索整个设计空间，并将上下文管理逻辑、工作流、权限以及许多其他 harness 组件一起优化。正如我们在 Meta-Harness、ADAS 和 AFlow 等工作中所看到的，✨代码✨是定义程序和系统的通用语言。简而言之，harness 就是一段代码，它编排提示词、工具调用、子智能体、控制流、记忆和工作流逻辑如何协同运作。如果 LLM 能够优化执行智能体的代码，它就能进入一个比手写提示词大得多的设计空间。
自教优化器（STOP；Zelikman 等人 2023）是递归式脚手架改进的早期范例之一。在步骤 $t=0$ 时，一个种子改进器 $I_0$ 接收一个初始解 $s$、一个效用函数 $u$ 以及一个黑盒语言模型 $M$，并返回一个改进后的解 $s’$，即 $s’ = I(u, s; M)$。STOP 的目标并非直接改进 $s$，而是改进改进器 $I$ 本身。
首先，让我们将元效用定义为给定改进器函数 $I$ 在一组下游任务 $\mathcal{D}$ 上的平均效用：
$$ \hat{u}(I) \triangleq \frac{1}{\vert\mathcal{D}\vert}\mathbb{E}_{(u,s)\sim \mathcal{D}}[u(I(u,s; M))] $$
由于改进改进器函数本身就是一个优化问题，我们可以基于 $I_{t-1}$ 由元效用衡量的表现，通过自我改进更新递归地得到 $I_t$ 的新版本：
$$ I_t=I_{t-1}(\hat{u},I_{t-1};M) $$
在他们的实验中，改进后的改进器发现了多种策略，例如遗传算法、分解并改进各部分、多臂提示词老虎机、模拟退火、变化温度以及束搜索/树搜索。这类似于将 harness 工作流表示为一个可供优化的对象。
Zelikman et al.（2023）的研究结果中有一个警示性发现：STOP 在使用 GPT-4 时能够随着迭代提升平均下游性能，但在 GPT-3.5 和 Mixtral 等较弱模型上反而出现退化。仅靠递归结构是不够的。基础模型必须足够强大，才能改进这一机制。这意味着 harness 的改进能够使模型得到更好的部署，但智能仍然是核心。
Lin et al.（2026）更详细地研究了 harness 演化对模型能力的依赖关系。他们拆解出两个维度：（1）harness 更新指的是产出有用 harness 编辑的能力；（2）harness 收益指的是利用更新后的 harness 来更好地解决任务的能力。有趣的是，在他们的实验中，从 Qwen3.5-9B 到 Claude Opus 4.6 等一系列不同规模和核心智能水平的模型，都表现出相似的 harness 更新能力；9B 的 harness 提议者/演化者能够写出与 Opus 在程序上同构的技能。要最好地利用 harness，模型需要正确且及时地调用技能/工具，并擅长长时程指令遵循。
一项更近期的研究，Self-Harness（Zhang et al. 2026），依赖 LLM 智能体通过“提议-评估-接受”循环来改进自身的 harness。
Self-Harness 中的循环包含三个阶段：
- Weakness mining: cluster failures into verifier-grounded failure patterns.
- 使用当前 harness $h_t$ 在任务上进行评估，并收集执行轨迹用于分析。
- 需要注意的是，两次运行在错误日志表面上可能共享相同的验证器结果，例如超时或缺失产物，但其因果机制却不同。因此，我们需要一份信息丰富的失败记录，包含终端验证器层面的原因、相关智能体行为的因果状态，以及轨迹所揭示的抽象智能体机制，以揭示根本原因。
- Harness proposal: propose bounded harness edits based on mined failure patterns.
- 在 $h_t$ 下调用同一个模型作为提议者。
- 模型被提供一个有界的提议上下文：(1) 当前 harness 的可编辑表面，(2) 来自评估系统的、以验证器为依据的失败模式，(3) 应当保留的通过行为记录，以及 (4) 先前尝试过的编辑的摘要。
- harness 编辑应优先考虑可解决的反复出现的错误模式（例如，不是任务特定的难度），并且可以通过窄范围的改动来解决。
- Harness 编辑候选应当彼此不同且多样。
- Proposal validation: validate and merge qualified edits to create a new harness $h_{t+1}$.
- 候选编辑通过在留入集 $D_\text{in}$（用于测试弱点是否已解决）和留出集 $D_\text{out}$（用于检查是否引入了其他未知问题）划分上的回归测试来评估。
- 只有当候选在留入集和留出集数据上均无回归时，才会被接受。
- 被接受的候选会被合并，以将 harness 更新为 $h_{t+1}$，而被拒绝的候选则会被记录，但不改变当前生效的 harness。
在 Terminal-Bench-2 上运行 MiniMax M2.5、Qwen3.5-35B-A3B 和 GLM-5 时，Self-Harness 被证明能够学习到针对不同基础模型不同弱点的模型专属 harness 指令，并提升留出集的通过率。
Self-harness 这类工作确实引发了我的担忧：如果允许程序编辑操作系统，抽象边界就会被打破。可编辑面需要被妥善设计，权限控制和安全层需要位于这个循环之外。围绕 reward hacking 的所有挑战依然存在。
Agentic Harness Engineering（AHE；Lin et al. 2026）认为 harness 演进的瓶颈在于 可观测性——也就是说，当一次 rollout 失败时，我们需要知道是哪个组件对此负责，并且每一次编辑都应当有证据支撑。
该框架构建了一个闭环，包含 3 个可观测性支柱：
- Component observability: every editable harness component has a representation in the file system so the action space is explicit and tracable.
- 一个 harness 包含 7 个组件：系统提示词、工具描述、工具实现、中间件、技能、子智能体配置和长期记忆。
- 每种失败模式都映射到一个组件，从而使编辑更具针对性。
- Experience observability: analysize and summarize a large amount of raw trajectories into a hierarchy of evidence and failure patterns.
- 每个 harness 生成 $k$ 条轨迹。
- 使用一个智能体（“Agent debugger”）来分析各自存储在一个文件中的轨迹，并针对每个任务生成关于失败或成功根因的分析报告。
- 所有按任务划分的报告会被汇总为一份基准概览，供下一步使用；如有需要，也可以访问原始轨迹。这种分层访问结构在 token 使用上更高效。
- Decision observability: every edit is paired with a prediction for the next round to validate.
- 一个智能体（“Evolve agent”）读取仓库，决定要编辑哪个组件，然后产出编辑内容及其背后的推理。
- Every edit is a file-level, falsifiable claim and can be verified in the next round, under two constraints:
- （1）编辑仅应用于 harness 工作区。runs 目录、tracer、verifier 和 LLM 配置均为只读，这消除了一系列奖励黑客行为（例如禁用 verifier、替换模型或提高推理预算），因此可以确保每一项记录在案的收益都可归因于 harness 编辑。
- (2) 编辑是证据驱动的，并附有一条宣言式条目：失败证据的名称、推断出的根因、针对性的修复方案，以及预测的影响，其中既包括预期修复的问题，也包括可能面临风险的回归。
在 Terminal-Bench-2 上，AHE 取得了优于人工设计的 harness（OpenCode、Terminus-2、Codex）的成绩，但 Hard 层级以及少数其他自进化基线（ACE、TF-GRPO）除外。同一个冻结的 harness，在不再进一步进化的情况下，迁移到了 SWE-bench-verified 上，这表明进化后的 harness 能够将工程经验编码进 harness 组件中，而不是进行针对特定基准的优化。
进化搜索
进化搜索是一种受自然选择启发的优化方法（参见我早前关于进化算法的文章）。它通过变异一组解，并且只保留群体中那些“适应度”高的解，来对解进行进化。当 (1) 搜索空间庞大或形状怪异，且 (2) 难以直接用梯度进行优化但容易评估解时，进化搜索就派上了用场。Harness 搜索似乎很适合这一方法。
进化搜索在过去的研究中已被用于提示词工程。Promptbreeder（Fernando 等，2023）通过丰富的变异操作来优化任务特定的提示词，有趣的是，变异提示词（即指示 LLM 对任务提示词进行变异的指令）本身也通过进化得到改进。GEPA（Agrawal 等，2025）将基于反思的提示词方法与进化搜索相结合，利用对试错轨迹的自然语言反思来提出提示词更新。
Novikov 等（2025）提出了 AlphaEvolve，作为一个编码智能体进化搜索系统，它维护一个候选程序池，并提示冻结的 LLM 生成用于改进的 diff。随着系统反复评估子程序并保留成功的程序，它随时间推移发现了更好的解决方案。
在 AlphaEvolve 的设计中，有几个细节至关重要：
- 提示词包含父程序、结果、指令，有时还包含元信息。
- 编码智能体可以访问完整的代码仓库，但待改进的代码区域会用
# EVOLVE-BLOCK-START和# EVOLVE-BLOCK-END显式标记。 - 元提示词与指令和上下文共同演化，正如 LLM 所建议的那样，其方式与我们演化求解程序的方式类似。
消融实验展示了演化过程、提示词中的上下文、元提示词、全文件演化以及使用更强 LLM 的作用。
近期的变体如 ThetaEvolve（Wang 等，2025）将演化搜索与 RL 和上下文学习相结合，而 DemoEvolve（Che 等，2026）则用人类专家演示来增强自 rollout 存档，作为用于 harness 级诊断和编辑的参考经验。另一方面，ShinkaEvolve（Lange 等，2025）引入了三个新组件来提升 LLM 采样效率：
- 通过设计父代采样来平衡性能排名和后代数量，从而实现更高效的样本探索。
- 基于嵌入向量的余弦相似度，丢弃与现有种群过于相似的候选方案，从而实现代码新颖性拒绝采样。
- 在元草稿本中识别成功方案中的良好模式，以指导未来的变异。
与上述聚焦于解决方案改进的方法不同，Darwin Gödel Machine（DGM；Zhang et al. 2025）明确以基于 LLM 的编程智能体来演化一个可编辑的 harness 代码仓库为目标。具体而言，该智能体被允许修改自身的 harness。后续关于 Hyperagents 的工作（Zhang et al. 2026）引入了一个元智能体，用来控制如何修改现有的任务智能体以创建新的智能体。
- 从池中的一个编程智能体开始。
- 在每次迭代中，按照与其性能成正比、与其已有子代数量成反比的概率选择一个父代，对其进行修改并分支以产生新的智能体。
- 被选中的父代智能体会检查自身的基准评测日志，然后对其自身的 harness 代码库提出改进，以生成新版本的编程智能体。代码编辑通过两个基本工具实现：（1）bash（参数：
<bash_command>）和（2）editor（参数：view/create/edit <file_path>）。 - 新的编程智能体会被评估，只有性能足够高的才会被加回池中。
- 重复步骤 2-4，直到满足某些停止条件。
DGM 是在固定模型下的 harness 演化。在以 Claude 3.5 Sonnet 作为基础 LLM 以及简单初始 harness 配置的实验中，DGM 发现的智能体在 SWE-bench Verified（20% 到 50%）和 Polyglot（14.2% 到 30.7%）上与手工设计的智能体相当或更优。
当候选解可以自动评估、且候选适应度易于量化时，这一系列方法效果很好，例如矩阵乘法、GPU kernel 优化、算法竞赛、数据中心调度。但在评估缓慢、模糊或主要依赖启发式的领域，它就会举步维艰。进化的计算效率与有效性也是值得关注的问题。
与模型权重联合优化
Harness 进化改变的是模型周围的非参数化系统。为了实现完全的自我改进，完全可以允许模型同时更新自己的权重。权重更新可以通过改进模型训练流程或测试时的持续学习来实现。持续学习这一主题值得未来单独写一篇文章。
SIA（Hebbar 等，2026）是将 harness 改进与模型参数更新结合在同一优化循环中的早期尝试，其设计包含三个组件：
- Meta-Agent：提出初始 harness。
- Task-Specific Agent：执行任务。
- Feedback-Agent：根据近期的轨迹决定是更新 harness 还是更新模型权重。
SIA 的实验中有一些混杂的选择，使得结果难以解读。例如，任务专用智能体比用于 Meta-Agent 和 Feedback-Agent 的模型弱得多（gpt-oss-120b 对 Claude Sonnet 4.6），而且基线太弱，无法与相关方法进行清晰的交叉对比。我认为这个方向很有趣，但证据尚属初步。然而，许多挑战，例如训练稳定性和 Goodhart 效应，仍然悬而未决。
Continual Harness（Karten et al. 2026）在长时程游戏环境中进行了实验，通过 harness 更新，并借助在低奖励轨迹上蒸馏强教师模型的标签来协同学习策略模型。
未来挑战
AI Scientist 这一系列工作有力地证明了，专家设计的 harness 可以协调自动研究循环中的很大一部分，并以撰写研究论文的形式进行了实验。但论文产出并不等同于科学发现。一个系统可以写出一篇看似合理的手稿，却仍然存在捏造的引用、实现漂移或实验结论薄弱的问题。
Trehan & Chopra（2026）测试了 LLM 能否在极少脚手架和基础工具（即 read_file、write_file、llm_search、list_files）的条件下，从一个研究想法走到一篇论文。每个想法都有一个专属工作区，智能体可以在其中生成和读取文档，作为上下文的一部分。他们在三个领域（世界模型、多智能体 RL、AI 安全与对齐）进行了实验，每个领域包含 45-50 篇高质量种子文档，用于激发新想法。只有四个想法被人类专家选中进入完整流程，而只有一个被完整执行成一篇论文。他们在实验中观察到六种反复出现的失败模式：
- 偏向训练数据默认值：使用旧库、过时命令、标准格式，或并非基于实际代码仓库或数据集的假设。
- 执行压力下的实现漂移：当实现变得技术上复杂时，模型可能会转向一种常见的更简单方案，而不是所提出的方法。
- 记忆与上下文退化：长周期项目会丢失关键细节，除非日志被写成持久化产物。
- 过度乐观：即便实验噪声很大或已经失败，模型仍宣称成功，这与 Bubeck et al.（2025）观察到的“p-hacking 和 eureka-ing”模式类似，即模型可能引入“数值胶带”，并在信号仍是噪声时宣布胜利。
- 领域智能不足：模型缺乏隐性的工艺知识，例如预测实现复杂度、判断某个实验结果是否合理，或知道哪些基线才重要。
- 科学品味薄弱：实验或许可以执行，但无法回答正确的问题。
在迈向完全 RSI 的道路上，研究者已经取得了切实的进展，但仍有若干瓶颈存在。
1. 评估器薄弱且模糊。许多研究主张没有快速而精确的验证器，许多现实世界任务同样如此。当前的自改进循环在评估指标可度量且客观的任务上效果最好，这与RL 的工作方式类似。
研究品味、新颖性和长期科学价值则要难衡量得多。例如，研究品味往往混合了问题界定、实验设计，以及对哪些意外结果值得继续追求、哪些失败案例值得重试的判断。
2. 上下文与记忆的生命周期。随着 AI 智能体变得更加自主和独立，记忆也会不断增长。一个有用的 harness 需要管理上下文和记忆，以弥补长上下文生成中现有的局限，同时仍然最大化长时程任务的成功率。既然人类能够在一生中维持记忆，我认为这里有一个类比：上下文工程将会也应当成为智能的核心组成部分，而不是停留在软件系统层。
3. 负面结果。 研究人员受到激励去发表成功的结果，因此文献对成功存在偏向。大语言模型在大量数据（至少目前大多由人类创造，哈哈）上训练，可能因为数据中成功与失败案例的不平衡，而不擅长判断何时该放弃一个假设、报告一个负面结果，甚至承认失败。一个研究框架应当让失败的尝试易于保存，因为从失败中学习是缩小任务搜索空间的最佳方式。
4. 多样性坍缩。 进化和强化学习循环倾向于利用已知的高奖励模式。我们需要机制来防止种群坍缩为同一解决方案的变体。这对于开放式研究尤为关键，因为在当前评估器下，最佳路径最初可能看起来更差。
5. 奖励黑客。 自我改进循环会优化它所获得的任何信号。如果奖励来自单元测试，智能体可能会过拟合到测试；如果奖励来自评判模型，它可能会学到针对该评判模型的特定奖励黑客技巧；如果奖励来自基准分数，它可能会利用基准的伪影。
评估器和权限控制很可能应当置于演化研究框架的循环之外，并配备留出测试、轨迹审计，以及在关键决策点上进行人工审查——监督能在多大程度上扩展和自动化，仍是一个开放的研究领域。
6. 长期成功。 一个外在的优化循环作用于单次 rollout 之外的奖励，这些奖励我们可以在训练沙盒中模拟。
以编码智能体为例。编码智能体已经提升了软件工程的日常生产力，但许多优化目标仍然过于短期。它通常能完成手头的任务，但如何保护一个由数百或数千名工程师共同维护的代码仓库的长期健康，则不那么显而易见。标准的基于沙盒的 RLVR 式训练很少能捕捉到可维护性、所有权边界、迁移成本、向后兼容性或未来的调试负担。
7. 人类的角色。 人类应当向上迁移到更高的层级，而不是被移出循环，这意味着人类应当在恰当的时机、恰当的抽象层级上提供监督，而我们的系统设计应当考虑何时以及如何设置这样的接触点。
上面列出的许多挑战都需要人类的反馈和引导。毕竟，我们是在为人类更美好的未来而构建技术，而不是反过来。
引用
请按如下方式引用本作品：
Weng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/
或使用 BibTeX 引用：
@article{weng2026harness,
title = {Harness Engineering for Self-Improvement},
author = {Weng, Lilian},
journal = {lilianweng.github.io},
year = {2026},
month = {July},
url = "https://lilianweng.github.io/posts/2026-07-04-harness/"
}
附录：一些有用的基准测试
- PaperBench: replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments.
- 每个复现任务都被分解为更小的、可单独评分的任务。
- 总共 8,316 条评分细则，与论文作者共同开发。
- 当时最好的模型（
Claude 3.5 Sonnet，约 21%）并未超过机器学习博士。 - 包含 PaperBench、PaperBench Code-Dev（轻量版）以及 JudgeEval。
- CORE-Bench: evaluate computational reproducibility of published research.
- 基于计算机科学、社会科学和医学领域的 90 篇科学论文，构建了 270 个任务。
- 任务涉及利用所提供的代码和数据复现结果。
- 包含多个难度级别，以及纯语言任务和视觉-语言任务。
- 当时报告的最佳智能体（
GPT-4o和GPT-4o-mini）在最难的任务上仅达到 21% 的准确率。
- ScienceAgentBench: evaluate LLM agents for data-driven scientific discovery.
- 从四个学科（数学、化学、生物学、地理学）的 44 篇同行评审出版物中提取了 102 个任务。
- 涵盖这些领域的基础数据科学任务：数据处理、模型开发、数据分析和信息可视化。
- RE-Bench: evaluate frontier AI agents on realistic ML research-engineering envs against human experts.
- 7 个具有挑战性的、开放式的机器学习研究工程环境。
- 每个环境 =（评分函数、起始解、参考解）；每个环境都可以在 8 块或更少的 H100 GPU 上运行。
- 示例：优化 kernel、运行缩放律实验、修复嵌入向量、为问答任务微调 GPT-2，等等。
- 包含来自 61 位不同人类专家的 71 次八小时尝试的数据。
- 人类专家在 82% 的 8 小时尝试中取得了非零分数；24% 达到或超过了强参考解。
- 在 2 小时预算下，最佳 AI 智能体的得分是人类专家的 4 倍，但人类在更长预算下的回报更高，并在 8 小时和 32 小时设置下超过了智能体。
- MLE-bench: evaluate ML engineering agents on offline Kaggle competitions.
- 包含从 Kaggle 精选的 75 个机器学习工程竞赛。
- 测试训练模型、准备数据集、运行实验，以及向评分脚本提交预测。
- 使用 Kaggle 公开排行榜作为人类基线。
- 论文中的最佳配置，
o1-preview搭配 AIDE 脚手架，在 16.9% 的竞赛中至少达到了 Kaggle 铜牌水平。 - 包含资源缩放和污染分析。
- KernelBench: evaluate correctness and speed for generated GPU kernels.
- 250 个 PyTorch 任务，用于评估 LLM 能否编写快速且正确的 kernel。
- 评估指标 fast_p = 生成的 kernel 中既正确又比基线更快的百分比。
参考文献
[1] Good, I. J. “关于第一台超智能机器的推测。” Advances in Computers, 6:31–88, 1965.
[2] Yudkowsky, Eliezer. “递归式自我改进。” LessWrong, 2008.
[3] Choi, et al. “用于代码修复的锚定自我对弈。” ICML 2026.
[4] Zhao, et al. “绝对零度：零数据的强化自我对弈推理。” arXiv preprint arXiv:2505.03335, 2025.
[5] Yuan, et al. “自我奖励语言模型。” arXiv preprint arXiv:2401.10020, 2024.
[6] Chen, et al. “自我对弈微调将弱语言模型转化为强语言模型。” ICML 2024.
[7] Zhang, et al. “智能体上下文工程：为自我改进语言模型演化上下文。” ICLR 2026.
[8] Ye, et al. “通过智能体技能演化的元上下文工程。” arXiv preprint arXiv:2601.21557, 2026.
[9] Lee 等，“Meta-Harness：模型 Harness 的端到端优化。” arXiv 预印本 arXiv:2603.28052，2026。
[10] Lu 等，“迈向 AI 研究的端到端自动化。” Nature，651:914–919，2026。
[11] Meng 等，“ScientistOne：通过证据链迈向人类水平的自主研究。” arXiv 预印本 arXiv:2605.26340，2026。
[12] Kulikov 等，“Autodata：一个用于创建高质量合成数据的智能体数据科学家。” arXiv 预印本 arXiv:2606.25996，2026。
[13] Hu、Lu 和 Clune。“智能体系统的自动化设计。” ICLR 2025。
[14] Madaan 等，“Self-Refine：借助自我反馈的迭代式精炼。” NeurIPS 2023。
[15] Zhang 等，“AFlow：自动化生成智能体工作流。” ICLR 2025。
[16] Zelikman 等，“自学优化器（STOP）：递归式自我改进的代码生成。” COLM 2024。
[17] Zhang 等，“Self-Harness：能够自我改进的 Harness。” arXiv 预印本 arXiv:2606.09498，2026。
[18] Fernando 等，“Promptbreeder：通过提示词进化实现自指式自我改进。” arXiv 预印本 arXiv:2309.16797，2023。
[19] Agrawal, A. 等，“GEPA：反思式提示词进化可超越强化学习。” arXiv 预印本 arXiv:2507.19457，2025。
[20] Novikov 等，“AlphaEvolve：面向科学与算法发现的编程智能体。” arXiv 预印本 arXiv:2506.13131，2025。
[21] Lange、Imajuku 和 Cetin，“ShinkaEvolve：迈向开放式且样本高效的程序进化。” arXiv 预印本 arXiv:2509.19349，2025。
[22] Wang 等，“ThetaEvolve：在开放问题上的测试时学习。” arXiv 预印本 arXiv:2511.23473，2025。
[23] Zhang 等，“Darwin Gödel Machine：自我改进智能体的开放式进化。” arXiv 预印本 arXiv:2505.22954，2025。
[24] Zhang 等，“Hyperagents。” arXiv 预印本 arXiv:2603.19461，2026。
[25] Yuksekgonul 等，“在测试时学习发现。” arXiv 预印本 arXiv:2601.16175，2026。
[26] Riaz 等，“面向测试时发现的认识不确定性。” arXiv 预印本 arXiv:2605.11328，2026。
[27] Hebbar 等，“SIA：通过 Harness 与权重更新实现自我改进的 AI。” arXiv 预印本 arXiv:2605.27276，2026。
[28] Trehan 与 Chopra。“为什么 LLM 还不是科学家：来自四次自主研究尝试的经验教训。” arXiv 预印本 arXiv:2601.03315，2026。
[29] Bubeck 等，“使用 GPT-5 的早期科学加速实验。” arXiv 预印本 arXiv:2511.16072，2025。
[30] Starace 等，“PaperBench：评估 AI 复现 AI 研究的能力。” ICML 2025。
[31] Wijk 等，“RE-Bench：在人类专家对照下评估语言模型智能体的前沿 AI 研发能力。” ICML 2025。
[32] Chan 等，“MLE-bench：在机器学习工程任务上评估机器学习智能体。” arXiv 预印本 arXiv:2410.07095，2024。
[33] Chen 等，“ScienceAgentBench：面向数据驱动科学发现的语言智能体的严格评估。” ICLR 2025。
[34] Siegel 等，“CORE-Bench：通过计算可复现性智能体基准促进已发表研究的可信度。” TMLR 2024。
[35] Ouyang 等，“KernelBench：LLM 能否编写高效的 GPU 内核？” arXiv 预印本 arXiv:2502.10517，2025。
[36] Lin 等，“Harness 更新不等于 Harness 收益：解耦自进化 LLM 智能体中的进化能力。” arXiv 预印本 arXiv:2605.30621，2026。
[37] Lin 等，“智能体 Harness 工程：可观测性驱动的编码智能体 Harness 自动进化。” arXiv 预印本 arXiv:2604.25850，2026。
[38] Karten 等，“持续 Harness：面向自我改进基础智能体的在线适应。” arXiv 预印本 arXiv:2605.09998，2026。
[39] Che 等，“DemoEvolve：利用演示克服智能体 Harness 进化中的稀疏反馈。” arXiv 预印本 arXiv:2605.24539，2026。
The concept of recursive self-improvement (RSI) dates back to I. J. Good (1965), where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence.
This feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the training pipeline and the deployment system, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (Anthropic; OpenAI).
I explicitly mention “deployment system” because the layer between the raw model and the real-world context seems to be as important as the model’s raw intelligence (i.e. the evals right after pretraining). Harnesses are important components of AI deployment, as shown by successful coding agent products such as Claude Code and Codex. A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results.
This one post will focus on research around harness engineering and how it contributes to RSI. Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post.
Harness Design Patterns
Compared with early agent frameworks, “agent = LLM + memory + tools + planning + action”, harnesses engineering additionally include workflow design (e.g. loop engineering), evaluation, permission controls, and persistent state management. It is no longer only prompt templates, but closer to runtime and software system design: how the model observes, acts, memorizes, checks itself, and improves.
The design should be deliberately simple and generic to enable generalization, likely with reference to existing software engineering practices to benefit from prertaining knowlege. There is also a strong analogy between operating systems and harnesses. Similar to an OS, a harness should encapsulate complicated logic while keeping the interface simple. Meanwhile, configs, tool interfaces and other protocols may gradually become standardized across the industry.
Pattern 1: Workflow Automation
Defining a workflow in which the model can operate, test, and iterate is a key design for automation. Karpathy’s autoresearch repo (https://github.com/karpathy/autoresearch) is a clean example of how such a workflow can be constructed. A common workflow follows a goal-oriented loop of plan, execute, observe/test, improve, and execute again until the goal is achieved. The process may trigger proactive requests to users for clarity in task specification or execution preference.
(Image source: OpenAI codex agent post)
The workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template.
Pattern 2: File System as Persistent Memory
A recurring pattern in long-horizon agent systems is simple control over rich states and artifacts. A harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files. In long-horizon agentic rollout, artifacts such as experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories often grow much longer than the context window that the model has trained for.
Learning how to read, write, and edit the file system (commonly via bash commands) is a foundation skill for LLMs, and thus managing persistent memory in the simple form of files naturally benefits from improvements in core model capability.
Pattern 3: Sub-agent and Backend Jobs
A harness can spawn multiple subagents to execute in parallel and monitor backend jobs. This is useful when the main agent needs to search multiple hypotheses, run experiments concurrently, or delegate isolated subtasks without polluting the main context. The parent agent then needs a small process manager: launch jobs, inspect logs, cancel failed runs, and merge results back into the main agent thread.
The key design choice is to make parallelism explicit and inspectable. If subagent outputs only live in a transient chat context, they quickly become obselete and hidden. If they are stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history.
Case study: Coding Agent Harness
The core interface of mainstream coding agents has become stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents. They commonly use a loop like:
With access to a set of tools, the coding agent is able to develop and debug issues in a given repository, similar to how human developers are equipped with IDEs.
(Not a comprenhensive list; shown for demonstration. Read this if interested.)
| Group | Tool definitions |
|---|---|
| File system | - File discovery: glob, grep, ls- File read: read, read_many- File modification: write (a whole new file); edit (string exact-match replacement); multi_edit; apply_patch (applies a structured patch/diff) |
| Shell execution | Run commands: bash, PowerShell |
| IO | lsp, git tools like git_status, git_diff, git_commit |
| External context | MCP tools, Skills |
| Web search | web_search, web_fetch, browser tools |
| Artifacts | Read docs, images; generate HTML, images |
| Backend processes | Such as: CronCreate, CronDelete, CronList |
| Agent delegation | Such as: spawn_agent, resume_agent, wait_agent, list_agents, close_agent, interrupt_agent, etc. |
Harness Layer vs Core Intelligence?
It is hard to forecast how much the future of RSI will rely on harness engineering, but the near-term path of RSI is unlikely to start as a model directly rewriting its weights. My prediction of a practical near-term path is:
- Harness engineering will evolve in the direction of meta-methodology (i.e. improving the machinery for getting better answers, not just improving the answer itself). The harness system itself becomes an optimization target, with fewer heuristic rules and more general mechanisms.
- In turn, mature harnesses enable auto-research for model self-improvement loop and smarter models prevents harnesses from overengineering and keep the system sustainable.
Eventually it is possible that many harness improvements will be internalized into core model behavior, but the interface with external context and tools should remain. We have seen a softer version of this pattern with prompt engineering: manual prompt tricks became less central as instruction tuning and model reasoning improved, but the need to specify goals, constraints, context, and evaluation did not disappear.
Harness Optimization
The progression in the object being optimized in the harness system is roughly: instruction prompts → structured context → workflow → harness code → optimizer code. As the model becomes more intelligent and powerful, we move toward more complex targets and generic methods.
Context Engineering
Simply appending all the tool responses and model generations into the context can quickly grow out of control as the agentic job horizon increases significantly. Context management is a layer to construct a more structed and concise context for LLM and manage persistant states. There is no doubt that long-context research will keep on making progress but at the moment long-context intelligence and context engineering sometime intertwines.
Agentic Context Engineering (ACE; Zhang et al. 2025) treats context as an evolving playbook rather than an increasingly lengthening prompt. It has three components to maintain one context playbook of bullet points, each with an identifier and a description.
- Generator: produces task trajectories, with reference to bullet points.
- Reflector: distills insights from successful and failed trajectories.
- Curator: updates the structured context with incremental, itemized entries.
To prevent context collapse and brevity bias during iterative rewrites, one key design choice in ACE is that the curator does not rewrite a full prompt blob. It instead outputs a collection of structured, itemized bullets in the form of (identifier, description), and these bullets are merged into a structured context logbook with deterministic logic. The context items are refined and deduplicated periodically.
The fact that ACE learns insights from rollouts helps us move toward self-managed memory, but the update rules and the overall workflow are still handcrafted. To move toward a more self-improving loop, Meta Context Engineering (MCE; Ye et al. 2026) separates the mechanism (how to manage context) from the artifact content (what is in context), running skill evolution at the meta-optimization level and context optimization at the base level.
An MCE skill $s \in \mathcal{S}$ defines a context function $c_s=(\rho_s,F_s)$ and maps an input $x$ to context $c = F_s(x;\rho_s)$, where:
- $\rho_s = \{\rho_1,\dots,\rho_m\}$ are static components (prompts, knowledge bases, code libraries).
- $F_s = \{F_1,\dots,F_k\}$ are dynamic operators (search, selection, filtering, formatting).
The bi-level optimization is to find the best context $c_s^*$ given skill $s$ on the training data, while the outer loop finds the optimal skill that provides the best performance on the validation set:
$$ \text{Inner: }c_s^*=\arg\max_{c_s}J_\text{train}(c_s;s)\quad \text{Outer: }s^*=\arg\max_{s\in\mathcal{S}}J_\text{val}(c_s^*) $$
The skill database tracks the history of previous skills, context functions and eval metrics $\mathcal{H}_{k-1} = \{(s_i,c_i,J_i^\text{train}, J_i^\text{val})\}_{i=1}^{k-1}$. A meta-level agent performs agentic crossover over prior skills to create a new skill given a task $\tau$: $s_k=\text{crossover}(\tau,\mathcal{H}_{k-1})$.
Then a base-level context engineer executes the skill $s_k$ and learns the context function from rollout feedback $\mathcal{R}_k$, guided by the current skill: $c_k=\text{engineer}(\tau,s_k;c_{k-1}^*,\mathcal{R}_k)$.
MCE does not enforce a heuristic rule for how to structure context as ACE does. It uses free-form skills to store the most important knowledge for a task, and evolves the skill and the skill-conditioned context iteratively together. Implementation-wise, a context function $c$ is instantiated as a collection of files in a dedicated directory, including both static (skill.md) and dynamic (context and data rollouts) components. Both meta-level and base-level optimization are executed in agentic coding envs with a standard tool set,
$$ \mathcal{T}=\{\texttt{Read},\texttt{Write},\texttt{Edit},\texttt{Bash},\texttt{Glob},\texttt{Grep},\texttt{TodoWrite}\} $$
Meta-Harness (Lee et al. 2026) moves another level deeper: the optimized object is the code that determines and optimizes what information should be stored, retrieved, and presented to the model. “Meta-” in its name means it is a harness for optimizing harnesses.
The proposer for creating a new harness is itself a coding agent and the final output is a collection of harness candidates on the Pareto frontier.
- The entire execution history is accessible via a file system, and thus the coding agent uses commands like
greporcatto read through it instead of shoveling everything into a single prompt context. - The proposed harness is a dictionary in the file system containing its own source code, scores, rollout trajectories, and state updates.
- The mete-harness loop iteratively creates new harnesses, and only qualified ones are kept.
Still, the important lesson is clear: once harness design becomes an executable search space, a strong coding agent can exploit the same design space human engineers use.
Workflow Design
Workflow design in harness engineering can be handcrafted by domain experts. Taking auto-research as an example, various frameworks have been proposed and tested. The AI Scientist system (Lu et al. 2026) builds a pipeline to propose research ideas, write code, run experiments, analyze results, write a manuscript, and perform peer review. Meng et al. (2026) make verifiability the central design constraint in ScientistOne, where every claim (citation, numerical, methodological, conclusion) must trace to an evidence source and is audited by Chain-of-Evidence checks.
The Autodata agent (Kulikov et al. 2026) is designed to work as a data scientist for generating training and evaluation data. The main agent manages a challenger that proposes problems, a weak solver, a strong solver, and a verifier/judge, aiming to synthesize data at the “just right” level of difficulty, meaning that the strong solver succeeds but the weak solver fails.
In Autodata, the challenger prompt is updated iteratively according to feedback from the solvers and verifier. The limitation here is that synthesized tasks are used to fine-tune weak solvers but not strong solvers; if the loop cannot iteratively improve the strong model, it is more like indirect distillation over a generated prompt distribution, with less RSI flavor.
The design space for workflow is enormous, and naturally we can think of workflow design as a search problem, and therefore we should be able to find good solutions by algorithms rather than only manually craft them. Following this direction, Automated Design of Agentic Systems (ADAS; Hu et al. 2025) formulates agent design itself as an optimization problem, “meta-agent search” where a meta-agent proposes new designs of agentic workflows.
- Initialize an archive of agentic workflows with simple agents such as CoT and self-refine.
- Ask a meta-agent to program new agents, all in code, inspired by existing solutions in the archive.
- The meta-agent first generates a high-level description of the new workflow, and then implements it in code.
- The draft program then goes through two self-refine steps (i.e. ask the model to provide feedback and then ask the same model to refine the previously generated outputs based on the feedback; Madaan et al. 2023) by the meta-agent to check its novelty.
- Evaluate each new candidate and add successful ones back to the archive.
- Repeat steps 2-3 until the maximum iteration count is reached.
(Image source: Hu et al. 2025)
AFlow (Zhang et al. 2025) represents an agentic workflow as a graph, where nodes represent LLM-invoking actions and edges implement logical operations in code. The workflow optimization relies on MCTS (Monte Carlo Tree Search):
- Initialize the starting workflow $W_0$ in the tree with a template.
- Select a workflow node using a soft mixture of score and uniform exploration.
- Expand it by asking an LLM to produce a modified workflow conditioned on its evaluation performance.
- Execute and evaluate the new workflow.
- Add it back to the tree if the new workflow shows improvement within a budget of $N$ rounds.
- Repeat steps 2-5 and stop when the top-$k$ average score plateaus or hit the budget.
Experiments of AFlow in QA, code, and math tasks showed decent improvement of AFlow over manually designed workflows and ADAS.
Self-Improving Harness
Either context engineering or workflow design is only one part of a harness. We need to search through the entire design space and optimize context-management logic, workflow, permissions, and many other harness components together. As we have seen in work like Meta-Harness, ADAS, and AFlow, ✨code✨ is a universal language for defining programs and systems. In simple words, a harness is code that programs how prompts, tool calls, subagents, control flow, memory, and workflow logic work together. If an LLM can optimize the code that executes agents, it can access a much larger design space than hand-written prompts.
Self-Taught Optimizer (STOP; Zelikman et al. 2023) is one of the early examples of recursive scaffolding improvement. A seed improver $I_0$ at step $t=0$ takes an initial solution $s$, a utility function $u$, and a black-box language model $M$, and returns an improved solution $s’$, that is, $s’ = I(u, s; M)$. The goal of STOP is not directly to improve $s$ but to improve the improver $I$ itself.
First, let’s define the meta-utility as the average utility of a given improver function $I$ over a collection of downstream tasks $\mathcal{D}$:
$$ \hat{u}(I) \triangleq \frac{1}{\vert\mathcal{D}\vert}\mathbb{E}_{(u,s)\sim \mathcal{D}}[u(I(u,s; M))] $$
Because improving the improver function is an optimization problem itself, we can recursively get a new version of $I_t$ based on $I_{t-1}$’s performance measured by meta-utility via a self-improvement update:
$$ I_t=I_{t-1}(\hat{u},I_{t-1};M) $$
In their experiments, the improved improver discovered various strategies, such as genetic algorithms, decomposing and improving parts, multi-armed prompt bandits, simulated annealing, varying temperature, and beam/tree search. This is analogous to how a harness workflow can be represented as an object for optimization.
A cautionary result in Zelikman et al. (2023)’s findings is that STOP improved mean downstream performance across iterations with GPT-4 but degraded with weaker models like GPT-3.5 and Mixtral. Recursive structure alone is not enough. The base model must be capable enough to improve the mechanism. This implies that harness improvement enables better deployment of the model but intelligence is still the core.
Lin et al. (2026) investigated the dependency of harness evolution on model capabilities in more details. They disentangled two axes: (1) harness-updating refers to the capability of producing useful harness edits and (2) harness-benefit denotes the capability of utilizing the updated harness, to achieve better task solving. Interestingly a range of model of different sizes and core intelligence, from Qwen3.5-9B to Claude Opus 4.6, were observed in their experiments to show similar harness updating capability; the 9B harness proposer/evolver is able to write a skill procedurally isomorphic to Opus. To best utilize a harness, a model needs to invoke skills/tools correctly and timely and be good at long-horizon instruction following.
A more recent work, Self-Harness (Zhang et al. 2026), relies on LLM agents to improve their own harness via a propose-evaluate-accept loop.
The loop in Self-Harness has three stages:
- Weakness mining: cluster failures into verifier-grounded failure patterns.
- The current harness $h_t$ is used to evaluate on tasks and execution traces are collected for analysis.
- Note that two runs can share the same verifier outcome in the error logs on the surface, such as timeout or missing artifact, while having different causal mechanisms. Therefore we need a failure record of rich information, containing the terminal verifier-level cause, the causal status of the relevant agent behavior, and the abstract agent mechanism exposed by the trace, to uncover the root causes.
- Harness proposal: propose bounded harness edits based on mined failure patterns.
- The same model is invoked under $h_t$ as a proposer.
- The model is provided with a bounded proposal context: (1) the editable surfaces of the current harness, (2) the verifier-grounded failure patterns from the evaluation system, (3) records of passing behaviors that should be preserved, and (4) summaries of previously attempted edits.
- Harness edits should prefer recurrent error patterns that are addressable (e.g. not task-specific difficulty) and can be resolved by narrow changes.
- Harness edit candidates should be distinct and diverse.
- Proposal validation: validate and merge qualified edits to create a new harness $h_{t+1}$.
- Candidate edits are evaluated by regression tests on held-in $D_\text{in}$ (for testing whether the weakness is resolved) and held-out $D_\text{out}$ (for checking whether other unknown issues were introduced) splits.
- Candidates are accepted only if they have no regression on both held-in and held-out data.
- Accepted candidates are merged to update the harness to $h_{t+1}$, while rejected candidates are logged without changing the active harness.
When running MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 on Terminal-Bench-2, Self-Harness was shown to learn model-specific harness instructions that target at different weaknesses of different base models and improve held-out pass rates.
Self-harness type of work does raise my concerns that if a program is allowed to edit the OS system, abstraction boundaries are broken. The editable surface needs to be properly designed and the permission control and security layers need to live outside this loop. All the challenges around reward hacking still remain.
Agentic Harness Engineering (AHE; Lin et al. 2026) see the bottlenecks of harness evolution are around observability—that is, when a rollout fails, we need to know which component is responsible for that and every edit should be grounded by evidence.
The framework creates a closed loop with 3 observability pillars:
- Component observability: every editable harness component has a representation in the file system so the action space is explicit and tracable.
- A harness contains 7 components: system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory.
- Each failure pattern is mapped to one component so the edit can be more targeted.
- Experience observability: analysize and summarize a large amount of raw trajectories into a hierarchy of evidence and failure patterns.
- Each harness generates $k$ traces.
- Use an agent (“Agent debugger”) to analysis the trajectories each stored in one file and generate per-task analysis report on the root cause for the failure or success.
- All the per-task reports are aggregated into a benchmark overview for the next step, and raw traces can be accessed if needed. This layered access structure is more token efficient.
- Decision observability: every edit is paired with a prediction for the next round to validate.
- An agent (“Evolve agent”) reads the repo and decides which component to edit, and then produces the edit and the reasoning behind it.
- Every edit is a file-level, falsifiable claim and can be verified in the next round, under two constraints:
- (1) Edits are only applied to the harness workspace. the runs directory, tracer, verifier, and LLM configuration are read-only, which disables a set of reward hacking (e.g disabling the verifier, swapping the model, or raising the reasoning budget) and thus it can keep every recorded gain attributable to harness edits.
- (2) Edits are evidence-driven, with a manifesto entry: the failure evidence’s name, the inferred root cause, the targeted fix, and a predicted impact comprising both expected fixes and at-risk regressions.
On Terminal-Bench-2, AHE achieved better than human-designed harness (OpenCode, Terminus-2, Codex) except for Hard tier and a few other self-evolve baselines (ACE, TF-GRPO). The same frozen harness, without further evolving, transfers to SWE-bench-verified, indicating that the evolved harness is able to encode engineering experience into harness components rather than doing benchmark-specific optimization.
Evolutionary Search
Evolutionary search is an optimization method inspired by natural selection (see my old post on evolutionary algorithm). It evolves a population of solutions by mutating them and only keeping those with high “fitness” in the crowd. Evolutionary search comes in handy when (1) the search space is extensive or weirdly shaped; and (2) it is hard to optimize directly with gradients but easy to evaluate solutions. Harness search seems to be a good fit here.
Evolutionary search has been used in prompt engineering in the past studies. Promptbreeder (Fernando et al. 2023) optimizes task-specific prompts through a rich set of mutation operations, and interestingly the mutation prompts (i.e. instructions to an LLM to mutate a task prompt) are themselves also improved through evolution. GEPA (Agrawal et al. 2025) combines reflection-based prompting with evolutionary search and uses natural language reflection over trajectories of trial and error to propose prompt updates.
Novikov et al. (2025) introduced AlphaEvolve as a coding-agent evolutionary search system, which stores a pool of candidate programs and prompts frozen LLMs to generate diffs for improvement. As the system repeatedly evaluates child programs and keeps successful ones, it discovers better solutions in time.
A few details matter in the design of AlphaEvolve:
- The prompt includes parent programs, results, instructions, and sometimes meta information.
- The coding agent has access to the full repo, but code regions for improvement are explicitly marked with
# EVOLVE-BLOCK-STARTand# EVOLVE-BLOCK-END. - Meta-prompt co-evolves with instructions and context as suggested by LLM, in a similar way as how we evolve solution programs.
Ablations show the evolution procedure, context in prompts, meta-prompts, full-file evolution and the use of stronger LLMs.
Recent variants such as ThetaEvolve (Wang et al. 2025) combines evolutionary search with RL and in-context learning, and DemoEvolve (Che, et al. 2026) augments the self-rollout archive with human expert demonstrations as reference experience for harness-level diagnosis and editing. ShinkaEvolve (Lange et al. 2025), on the other hand, introduced three new components to improve LLM sampling efficiency:
- More sample-efficient exploration by designing parent sampling to balance performance rank and offspring count.
- Code-novelty rejection sampling by discarding candidates that are too similar to the existing population based on embedding-based cosine similarity.
- Identifying good patterns in successful solutions in a meta-scratchpad to guide future mutation.
Unlike the methods above, which focus on solution improvement, Darwin Gödel Machine (DGM; Zhang et al. 2025) explicitly targets the evolution of an editable harness-code repository with an LLM-based coding agent. Precisely, this agent is allowed to modify its own harness. A follow-up work on Hyperagents (Zhang et al. 2026) introduced a meta-agent to control how to modify existing task agents to create new ones.
- Start with one coding agent in the pool.
- In each iteration, pick one parent with a probability proportional to its performance and inversely to the number of children it has, to modify and branch off to produce new agents.
- The selected parent agent examines its own benchmark evaluation log and then proposes improvements to its own harness codebase to generate a new version of the coding agent. Code editing is implemented with two basic tools: (1) bash (args:
<bash_command>) and (2) editor (args:view/create/edit <file_path>). - New coding agents are evaluated, and only those with sufficiently high performance are added back into the pool.
- Repeat steps 2-4 until some stop criteria hit.
DGM is harness evolution under a fixed model. In experiments with Claude 3.5 Sonnet as the base LLM and simple initial harness configs, the DGM-discovered agents are comparable to or outperform handcrafted agents on SWE-bench Verified (20% to 50%) and Polyglot (14.2% to 30.7%).
This family of methods works well when candidate solutions are automatically evaluable and candidate fitness is easy to quantify, such as matrix multiplication, GPU kernel optimization, algorithm contests, datacenter scheduling. It struggles with domains where evaluation is slow, ambiguous, or mostly heuristic-based. The compute efficiency and effectiveness of evolution are also concerns.
Joint Optimization with Model Weights
Harness evolution changes the non-parametric system around the model. To enable full self-improvement, the model can totally be allowed to update its own weights at the same time. The weight update can be implemented via improvements in the model training pipeline or continual learning at test time. The topic of continual learning is worthy of its own post in the future.
SIA (Hebbar et al. 2026) is an early attempt to combine harness improvement and model-parameter updates in the same optimization loop, with three components in the design:
- Meta-Agent: proposes the initial harness.
- Task-Specific Agent: executes the task.
- Feedback-Agent: chooses whether to update the harness or the model weights based on recent trajectories.
There are a few confounding choices in SIA’s experiments that make the results hard to interpret. For example, the task-specific agent is much weaker than the models used for the Meta-Agent and Feedback-Agent (gpt-oss-120b vs Claude Sonnet 4.6), and the baselines are too weak to cross-reference cleanly against related methods. I would consider the direction interesting, but the evidence provisional. Yet many challenges, such as training stability and Goodhart effect, still remain open.
Continual Harness (Karten et al. 2026) experimented in long-horizon gameplay setting with harness updating and co-learning a policy model by distilling a strong teacher model’s labels on low-reward trajectories.
Future Challenges
The AI Scientist line of work is a strong demonstration that an expert-designed harness can coordinate a large portion of auto-research loop, experimented in the form of writing research papers. But paper production is not identical to scientific discovery. A system can write a plausible manuscript while still having fabricated citations, implementation drift, or weak experimental results.
Trehan & Chopra (2026) tested whether LLMs can go from a research idea to a paper with minimal scaffolding and basic tools (i.e., read_file, write_file, llm_search, list_files). Each idea had a dedicated workspace where agents could generate and read documents as part of context. They experimented in three domains (world models, multi-agent RL, AI safety & alignment), with each domain containing 45-50 high-quality seed documents to inspire new ideas. Only four ideas were selected by human experts to run through the full pipeline, and only one was fully executed into a paper. They observed six recurring failure modes in the experiments:
- Bias toward training-data defaults: use old libraries, stale commands, standard formats, or assumptions not grounded in the actual repository or dataset.
- Implementation drift under execution pressure: when implementation becomes technically complex, the model may move toward a common simpler solution rather than the proposed method.
- Memory and context degradation: long-horizon projects lose critical details unless logs are written as persistent artifacts.
- Over-optimism: the model declares success despite noisy or failed experiments, similarly observed as “p-hacking and eureka-ing” pattern by Bubeck et al. (2025) where models can introduce “numerical duct tape” and declare victory when signals are still noise.
- Insufficient domain intelligence: the model lacks tacit craft knowledge, e.g. predicting implementation complexity, judging whether an experimental result is plausible, or knowing which baselines matter.
- Weak scientific taste: experiments may be executable but fail to answer the right question.
Toward full RSI, researchers have made real progress, but several bottlenecks remain.
1. Weak and fuzzy evaluators. Many research claims do not have a fast and precise verifier, and the same is true for many real-world tasks. Current self-improvement loops work best for tasks when evaluation metrics are measurable and objective, similar as how RL works.
Research taste, novelty, and long-term scientific value are much harder to measure. For example, research taste often mixes problem framing, experimental design, and judgment about which surprising results are worth pursuing and which failure cases are worth retries.
2. Context and memory lifecycle. Memory grows as AI agents become more autonomous and independent. A useful harness needs to manage context and memory to complement existing limitation in long-context generation while still maximizing the success of long-horizon tasks. Since humans are able to maintain memory through our life time, I see an anoloy here that context engineering will and should become a core part of intelligence, rather than staying in the software system layer.
3. Negative results. Researchers are incentivized to publish successful results and thus literature is biased toward successes. LLMs trained on a vast amount of data (mostly human created, at least for now, lol) may be bad at deciding when to abandon a hypothesis, report a negative result, or even acknowledge a failure due to the imablance of success vs failure cases in data. A research harness should make failed attempts easy to preserve, as learning from failure is the best way to trim down the task search space.
4. Diversity collapse. Evolutionary and RL loops tend to exploit known high-reward patterns. We need mechanisms to prevent the population from collapsing into variants of the same solution. This is especially critical for open-ended research, where the best path may initially look worse under the current evaluator.
5. Reward hacking. A self-improvement loop optimizes whatever signal it is given. If the reward comes from unit tests, the agent may overfit to tests; if it comes from a judge model, it may learn reward hacking tricks specific to this judge; if it comes from benchmark scores, it may exploit benchmark artifacts.
The evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter—how much oversight can be scaled up and automated remains an open research area.
6. Long-term success. An extrinsic loop of optimization works on rewards outside of individual rollouts that we can simulate in training sandbox.
Take coding agent as an example. Coding agents have already increased daily productivity in software engineering, but many optimization goals are still too short-term. It can often complete the task at hand, but less obvious how it should protect the long-term health of a repo collectively maintained by hundreds or thousands of engineers. Standard sandbox-based RLVR-style training rarely captures maintainability, ownership boundaries, migration cost, backwards compatibility, or future debugging burden.
7. The role of humans. Humans should move up the stack, not be removed from the loop, meaning that human should provide oversight at the right time, at the right abstraction level and our system design should consider when and how to set up such touch points.
Many challenges listed above need human’s feedback and steering. After all, we are building the technology for better future of humanity, not other way around.
Citation
Please cite this work as:
Weng, Lilian. “Harness Engineering for Self-Improvement”. Lil’Log (Jul 2026). https://lilianweng.github.io/posts/2026-07-04-harness/
Or use the BibTeX citation:
@article{weng2026harness,
title = {Harness Engineering for Self-Improvement},
author = {Weng, Lilian},
journal = {lilianweng.github.io},
year = {2026},
month = {July},
url = "https://lilianweng.github.io/posts/2026-07-04-harness/"
}
Appendix: Some useful benchmarks
- PaperBench: replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments.
- Each replication task is decomposed into smaller, individually gradable tasks.
- 8,316 rubrics in total, co-developed with the paper authors.
- The best model at the time (
Claude 3.5 Sonnet, ~21%) does not outperform ML PhDs. - Includes PaperBench, PaperBench Code-Dev (a lighter version), and JudgeEval.
- CORE-Bench: evaluate computational reproducibility of published research.
- 270 tasks based on 90 scientific papers across computer science, social science, and medicine.
- Tasks involve reproducing results from provided code and data.
- Includes multiple difficulty levels and both language-only and vision-language tasks.
- The best reported agent at the time (
GPT-4oandGPT-4o-mini) achieved only 21% accuracy on the hardest task.
- ScienceAgentBench: evaluate LLM agents for data-driven scientific discovery.
- Extracts 102 tasks from 44 peer-reviewed publications in four disciplines (math, chemistry, biology, geography).
- Covers basic data-science tasks in these domains: data processing, model development, data analysis, and information visualization.
- RE-Bench: evaluate frontier AI agents on realistic ML research-engineering envs against human experts.
- 7 challenging, open-ended ML research-engineering environments.
- Each environment = (scoring function, starting solution, reference solution); each can be run with 8 or fewer H100 GPUs.
- Examples: optimize a kernel, run a scaling-law experiment, fix an embedding, fine-tune GPT-2 for QA, etc.
- Includes data from 71 eight-hour attempts by 61 distinct human experts.
- Human experts achieved non-zero score in 82% of 8-hour attempts; 24% matched or exceeded strong reference solutions.
- Best AI agents scored 4× higher than humans at a 2-hour budget, but humans had better returns to longer budgets and exceeded agents at 8-hour and 32-hour settings.
- MLE-bench: evaluate ML engineering agents on offline Kaggle competitions.
- Contains 75 ML-engineering competitions curated from Kaggle.
- Tests training models, preparing datasets, running experiments, and submitting predictions to grading scripts.
- Uses Kaggle public leaderboards as human baselines.
- Best setup in the paper,
o1-previewwith AIDE scaffolding, reached at least Kaggle bronze-medal level in 16.9% of competitions. - Includes resource-scaling and contamination analyses.
- KernelBench: evaluate correctness and speed for generated GPU kernels.
- 250 PyTorch tasks to evaluate whether LLM can write fast and correct kernels.
- The evaluation metric fast_p = the percentage of generated kernels that are correct and faster than baseline.
References
[1] Good, I. J. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers, 6:31–88, 1965.
[2] Yudkowsky, Eliezer. “Recursive Self-Improvement.” LessWrong, 2008.
[3] Choi, et al. “Anchored Self-Play for Code Repair.” ICML 2026.
[4] Zhao, et al. “Absolute Zero: Reinforced Self-play Reasoning with Zero Data.” arXiv preprint arXiv:2505.03335, 2025.
[5] Yuan, et al. “Self-Rewarding Language Models.” arXiv preprint arXiv:2401.10020, 2024.
[6] Chen, et al. “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.” ICML 2024.
[7] Zhang, et al. “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.” ICLR 2026.
[8] Ye, et al. “Meta Context Engineering via Agentic Skill Evolution.” arXiv preprint arXiv:2601.21557, 2026.
[9] Lee, et al. “Meta-Harness: End-to-End Optimization of Model Harnesses.” arXiv preprint arXiv:2603.28052, 2026.
[10] Lu, et al. “Towards end-to-end automation of AI research.” Nature, 651:914–919, 2026.
[11] Meng, et al. “ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.” arXiv preprint arXiv:2605.26340, 2026.
[12] Kulikov, et al. “Autodata: An agentic data scientist to create high quality synthetic data.” arXiv preprint arXiv:2606.25996, 2026.
[13] Hu, Lu, and Clune. “Automated Design of Agentic Systems.” ICLR 2025.
[14] Madaan, et al. “Self-Refine: Iterative Refinement with Self-Feedback.” NeurIPS 2023.
[15] Zhang, et al. “AFlow: Automating Agentic Workflow Generation.” ICLR 2025.
[16] Zelikman, et al. “Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation.” COLM 2024.
[17] Zhang, et al. “Self-Harness: Harnesses That Improve Themselves.” arXiv preprint arXiv:2606.09498, 2026.
[18] Fernando, et al. “Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution.” arXiv preprint arXiv:2309.16797, 2023.
[19] Agrawal, A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning.” arXiv preprint arXiv:2507.19457, 2025.
[20] Novikov, et al. “AlphaEvolve: A coding agent for scientific and algorithmic discovery.” arXiv preprint arXiv:2506.13131, 2025.
[21] Lange, Imajuku, and Cetin. “ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution.” arXiv preprint arXiv:2509.19349, 2025.
[22] Wang, et al. “ThetaEvolve: Test-time Learning on Open Problems.” arXiv preprint arXiv:2511.23473, 2025.
[23] Zhang, et al. “Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents.” arXiv preprint arXiv:2505.22954, 2025.
[24] Zhang, et al. “Hyperagents.” arXiv preprint arXiv:2603.19461, 2026.
[25] Yuksekgonul, et al. “Learning to Discover at Test Time.” arXiv preprint arXiv:2601.16175, 2026.
[26] Riaz, et al. “Epistemic Uncertainty for Test-Time Discovery.” arXiv preprint arXiv:2605.11328, 2026.
[27] Hebbar, et al. “SIA: Self Improving AI with Harness & Weight Updates.” arXiv preprint arXiv:2605.27276, 2026.
[28] Trehan and Chopra. “Why LLMs Aren’t Scientists Yet: Lessons from Four Autonomous Research Attempts.” arXiv preprint arXiv:2601.03315, 2026.
[29] Bubeck, et al. “Early science acceleration experiments with GPT-5.” arXiv preprint arXiv:2511.16072, 2025.
[30] Starace, et al. “PaperBench: Evaluating AI’s Ability to Replicate AI Research.” ICML 2025.
[31] Wijk, et al. “RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.” ICML 2025.
[32] Chan, et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.” arXiv preprint arXiv:2410.07095, 2024.
[33] Chen, et al. “ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.” ICLR 2025.
[34] Siegel, et al. “CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.” TMLR 2024.
[35] Ouyang, et al. “KernelBench: Can LLMs Write Efficient GPU Kernels?” arXiv preprint arXiv:2502.10517, 2025.
[36] Lin, et al. “Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents.” arXiv preprint arXiv:2605.30621, 2026.
[37] Lin, et al. “Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses.” arXiv preprint arXiv:2604.25850, 2026.
[38] Karten, et al. “Continual Harness: Online Adaptation for Self-Improving Foundation Agents.” arXiv preprint arXiv:2605.09998, 2026.
[39] Che, et al. “DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations.” arXiv preprint arXiv:2605.24539, 2026.