Xinyu Lu

Tianshu Wang

Pengbo Wang

Zujie Wen

Zhiqiang Zhang

Jun Zhou

Boxi Cao

Yaojie Lu

Hongyu Lin

Xianpei Han

Le Sun

Chinese Information Processing Laboratory,

Institute of Software, Chinese Academy of Sciences

University of Chinese Academy of Sciences

Ant Group

Work done during an internship at Ant Group.

Abstract

Current AI benchmarks evaluate agents on task execution within human-designed workflows. These evaluations fundamentally fail to measure a critical next-level capability: whether models can autonomously develop agent systems. We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development. Specifically, a code agent (the meta-agent) is given a sandboxed environment, an evaluation API, and a time limitation to iteratively program an agent artifact that maximizes performance on a held-out test set across five domains. To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking. Leveraging this framework, we demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models. Moreover, the design process exhibits high variance, and high optimization pressure surfaces emergent adversarial behaviors like ground-truth exfiltration—highlighting critical deficits in both robustness and model alignment. Ultimately, MAC provides a rigorous, open-source benchmark for autonomous AI research and development, offering an empirical proxy for evaluating recursive self-improvement. Benchmark is publicly available at: https://github.com/ant-research/meta-agent-challenge.

1 Introduction

Current Large Language Models (LLMs) are increasingly capable of performing long-horizon tasks [22, 8]. To tackle increasingly complex tasks, the community has developed sophisticated agentic scaffoldings [20]—augmenting raw model capabilities with external tools, iterative reflection, and complex subagent orchestration ability [7]. These frameworks have significantly expanded the boundaries of what AI systems can achieve, transforming stateless language models into autonomous and persistent task executors.

However, a fundamental limitation persists: these advanced agent scaffoldings are almost exclusively hand-crafted by human researchers and developers. The current paradigm of agent development relies heavily on manual prompt engineering, human-designed control flows, and human-defined tools. While existing benchmarks rigorously evaluate how well an LLM executes a specific task within these human-defined evaluation workflows, they fundamentally fail to measure a far more critical capability for the future of AI: the ability of the model to act as a system architect. Can an AI independently design, implement, evaluate, and iteratively optimize its own task-solving workflows?

We argue that overcoming the bottleneck of manual agent engineering requires a paradigm shift (Figure 1) in how we evaluate intelligence—moving from the object level of task execution to the meta level of system design. To this end, we introduce the Meta-Agent Challenge (MAC), a novel evaluation framework designed to test the capacity of current code agents for autonomous agent development. In this challenge, the evaluated model (the meta-agent) is not tasked with directly solving domain-specific problems. Instead, it is provided with an objective function, a model access quota, and a sandboxed development environment, and is challenged to write the underlying code to build task-specific agents that solves the problems.

This shift in evaluation formulation is crucial for several reasons. First, it bridges the gap between writing isolated snippets of code and engineering robust, autonomous software systems. A successful meta-agent must formulate hypotheses about effective architectures, implement them as executable programs, run empirical evaluations on a development set, diagnose failure modes from limited feedback, and iterate—all within a single autonomous session. Second, and more importantly, it serves as a concrete, actionable proxy for recursive self-improvement [16]. By closing the loop between agent evaluation and agent construction, our framework provides a rigorous testbed to measure whether models have reached the threshold where they can systematically improve AI systems without human intervention.

Moreover, measuring autonomous agent development capability of frontier models is important to AI safety [1]. It probes the degree to which a model can recognize its own capabilities and limitations, plan resource usage, and potentially construct more powerful systems. Additionally, the optimization pressure in our framework surfaces misalignment behaviors, offering a sandboxed setting for evaluating risks in real-world agentic systems.

To establish the first comprehensive evaluation suite for autonomous agent self-improvement, we make the following core contributions:

1. (1)

   We formalize and instantiate the Meta-Agent Challenge across diverse domains—mathematical reasoning, competitive programming, graduate-level science, repository-level software engineering, and long-horizon terminal interactions—releasing the first open-source evaluation benchmark for autonomous agent development.

2. (2)

   We design and implement a highly secure evaluation process with multi-layered countermeasures against reward hacking behavior of agents. While novel exploits may inevitably arise, our defensive architecture establishes a critical foundation for rigorous and trustworthy agent benchmarking under high optimization pressure.

3. (3)

   Leveraging this framework, we demonstrate that meta-agents rarely match human-engineered scaffolds and the few that do are dominated by proprietary models, revealing a substantial gap between closed- and open-source models. We further show that the autonomous design process exhibits high inter-run variance and that strong optimization pressure can trigger emergent misaligned behaviors. We then diagnose key success and failure modes to guide future agentic model development.

2 Related Works

Agent Benchmarks.

Existing agent benchmarks predominantly evaluate an agent’s ability to solve a specific class of tasks. For instance, SWE-Bench [6] measures repository-level code editing, while Terminal-Bench [11] targets long-horizon terminal interaction requiring sustained multi-step reasoning. BrowseComp [21] focuses on browsing agents that perform iterative long-chain retrieval over the web. Most closely related, MLE-Bench [3] evaluates the ability of agentic models to develop and optimize algorithms for machine learning tasks. A common thread across these benchmarks is their domain-specific nature: each measures how well an agent performs within a fixed task formulation. The Meta-Agent Challenge departs from this paradigm by evaluating not an agent’s task-solving ability, but its capacity to develop and iteratively improve agent systems across diverse domains.

Meta-Agents.

Many recent works have explored the concept of meta-agents – agent systems that design, configure, or optimize agent systems. For example, Hu et al. [4], Qiu et al. [13], Yin et al. [24], Xia et al. [23], Zhou et al. [27], Lee et al. [10] and Zhang et al. [25] focus on the design of meta-agent frameworks that orchestrate specialized agent systems. Meanwhile, some works utilize LLMs within evolutionary frameworks to autonomously propose, evaluate, and refine complex algorithms [12] or even the coding agent itself [26]. Concurrently, Lee et al. [10] find that modern code agents can perform optimization of agent harnesses. In contrast, our work does not propose a new meta-agent architecture; rather, we construct a framework to evaluate the potential of general-purpose code agents as meta-agents. In this sense, the Meta-Agent Challenge can be viewed as a dual of PostTrainBench [14] in the agent development domain: whereas PostTrainBench evaluates agents’ ability to conduct model post-training, our benchmark evaluates their ability to improve agent workflows themselves –closing the loop from agent evaluation to agent construction.

3 The Meta-Agent Challenge

We introduce the Meta-Agent Challenge, an evaluation framework for evaluating the capacity of code agents to autonomously design, implement, and iteratively optimize task-specific agent workflows. Unlike conventional benchmarks that assess an agent’s direct problem-solving ability, our challenge operates at a higher level of abstraction: the agent under evaluation does not solve problems itself, but instead constructs another agent that does. This recursive structure—an agent building an agent—tests capabilities that lie at the frontier of current systems: hypothesis formation, code implementation, empirical evaluation, and iterative refinement.

3.1 Formulation

Let denote a code agent (the meta-agent) with access to foundation model APIs, supplementary tool APIs (e.g., web search), and a sandboxed development environment. The meta-agent is tasked with producing an artifact —an executable program implementing a task-specific agent—that maximizes performance on a held-out test set .

Concretely, the meta-agent has access to:

1. A development set of domain-specific queries or tasks;

2. An evaluation endpoint that accepts an agent artifact and returns performance feedback on ;

3. A comprehensive API suite providing access to language models and external environmental tools (such as search engines) for use within the produced artifact;

4. A base class specification defining the agent interface required by the evaluation system.

Meanwhile, the meta-agent is constrained to:

1. API resource limitations and : A strictly enforced budget on the number of API calls, total tokens consumed allocated for both the development phase and the execution of the final artifact. This prevents trivial solutions that rely on excessive sampling or brute-force queries.

2. Time limitations and : A maximum computational time budget for the development phase and the artifact test phase, requiring the meta-agent to efficiently balance the exploration of novel agent architectures against the exploitation of known working strategies before the deadline.

Therefore, the Meta-Agent Challenge can be cast as a constrained optimization problem. The meta-agent acts as an autonomous optimizer searching through the space of valid agent programs defined by the base class . The objective is to find an optimal artifact that maximizes the evaluation metric on the unseen test set:

				(1)
	s.t.

Crucially, because the target distribution is completely hidden during the development phase, the meta-agent cannot solve this optimization directly. Instead, it must rely on empirical feedback from to iteratively propose, evaluate, and refine , mirroring the trial-and-error development cycle of a human developer.

3.2 Evaluation Protocol

The evaluation of follows a two-phase protocol designed to mirror realistic agent development workflows while maintaining rigorous held-out evaluation.

Development Phase.

During the development phase, the meta agent may: (i) read the task instructions, (ii) implement and revise its agent artifact, (iii) submit the artifact to the evaluation oracle for feedback on , and (optionally) install additional dependencies. The evaluation API returns information such as per-problem correctness indicators and overall accuracy as feedback, enabling iterative optimization while preserving the integrity of the final evaluation on the held-out test set.

Verification Phase.

After the development budget expires, an automated verifier executes the following pipeline: (1) the API usage monitor scans the workspace for policy violations; (2) the agent artifact is executed on the held-out test set with a timeout of seconds; (3) predictions are graded against ground truth or unit tests; and (4) the final reward is recorded. The verifier script is injected into the container only at this stage, ensuring the meta-agent cannot access the test-split secret during development.

3.3 Evaluation System Architecture

A central design challenge of evaluating meta-agent capability is providing the agents with meaningful evaluation feedback while preventing it from trivially gaming the evaluation system. We address this through a dual-container architecture that enforces strict isolation between the agent execution environment and the evaluation service (Figure 2).

媒体内容 · 前往原文查看

Agent Container.

The agent container provides a sandboxed Linux environment in which the meta-agent develops its solution. It contains the agent interface specification (base_agent.py), an API wrapper (openai_helper.py). The meta-agent writes its artifact to /workspace/agent.py and may declare additional Python dependencies via a pyproject.toml manifest, which the evaluation system installs automatically.

Agent Interface.

All agent artifacts must inherit from BaseAgent class and implement a single method:

(2)

where is the problem index, is the problem statement, is the predicted answer, and is the timeout budget in seconds. The base class provides standard interface specification and utility methods. This minimal interface grants the meta-agent maximum flexibility in designing the internal architecture of its artifact—from simple single-pass prompting to sophisticated multi-stage pipelines.

Evaluation Container.

A separate container hosts a Flask-based evaluation service that holds all ground-truth answers for both splits. When the meta-agent submits its artifact via the /evaluate/agent endpoint, the evaluation container: (1) dynamically loads the submitted agent class, (2) executes it on the requested problem split, (3) grades predictions using domain-specific grading logic, and (4) returns performance feedback. This design ensures that individual ground-truth answers are never exposed to the agent container.

3.4 Evaluation Integrity

During the development process of the evaluation framework, we identify two primary threat modes that agents might exploit to artificially inflate their scores and address them through a multi-layered defense architecture.

Threat 1: Unauthorized Resource Access.

A meta-agent could bypass challenge constraints by tampering with API request parameters to clandestinely invoke a stronger, unauthorized model in same model API provider. Furthermore, it might circumvent resource quotas entirely by utilizing leaked external API keys or executing local language models. Both capability escalation and quota evasion render performance comparisons meaningless.

Mitigation. We enforce resource isolation through:

• API monitoring: A static analysis module scans all Python files in the agent workspace for unauthorized imports and network connections to disallowed endpoints.

• API proxy: All model API calls are routed through a proxy in the evaluation container that enforces quota limits and logs usage.

• Post-hoc auditing: An auditing agent analyzes execution traces and submitted code to identify unauthorized network requests, hidden model invocations, or attempts to bypass the API proxy.

Threat 2: Test Set Leakage.

Without proper isolation, a meta-agent could directly access test set problems or ground-truth answers, enabling trivial solutions that memorize answers rather than develop generalizable agent architectures.

Mitigation. We enforce strict data isolation through:

• Container filesystem separation: Ground-truth answers for both and reside exclusively in the evaluation container’s private filesystem, inaccessible from the agent container.

• Split-level access control: Evaluation on the test split requires a cryptographic secret transmitted via HTTP header (X-Verifier-Secret). This secret is embedded in the verifier script, which is injected into the agent container only after the development phase concludes. During development, the meta-agent can only evaluate against .

• Post-hoc auditing: The auditing agent detects hardcoded answers, brute-force enumeration attempts, and other patterns indicative of test set exploitation.

3.5 Domain Instantiations

The Meta-Agent Challenge framework is domain-agnostic by design. To demonstrate its generality and provide a comprehensive evaluation benchmark, we instantiate it across five domains that exercise complementary meta-agent capabilities: mathematical reasoning (AIME), graduate-level science QA (GPQA/HLE), competitive programming (LiveCodeBench), repository-level code editing (SWE-Bench), and long-horizon terminal interaction (Terminal-Bench). We refer to this evaluation suite as MAC-v1. Comprehensive details regarding the datasets, evaluation metrics, and specific split configurations for each domain are provided in Appendix C.

4 Experimental Setup

Meta-Agent Systems.

We evaluate four CLI-based autonomous coding agents driven by proprietary frontier models: Claude Code (Claude Opus 4.7, Opus 4.6, and Sonnet 4.6), Gemini-Cli (Gemini 3.1 Pro), Codex (gpt-5.3-codex, gpt-5.4), all of which operate within the Harbor [17] sandboxed evaluation harness. We also evaluate several leading open-weight models (GLM, Kimi, DeepSeek, MiniMax) integrated with the Claude Code scaffolding. Thanks to the highly extensible design of the Harbor framework, our evaluation system can seamlessly integrate any alternative agent scaffolding not explicitly used in experiments.

Resource Constraints.

For AIME, GPQA, and LiveCodeBench, the meta-agent is allocated a time budget of seconds (12 hours); for SWE-Bench and Terminal-Bench, seconds (24 hours), reflecting the higher cost of executing and evaluating full agent artifacts. In all domains the meta-agent has access to the evaluation API for iterative feedback on . Regarding the API quota , we design an evaluation setting for the first three domains utilizing a dedicated vLLM backend, where each evaluation instance has exclusive access to a model deployed via vLLM [9] on a single A100 GPU. For SWE-Bench and Terminal-Bench, all runs use commercial API endpoints exclusively, as execution and grading are delegated to the Harbor framework. Furthermore, the meta-agent is granted a search API quota of calls per phase in the science domain.

Baselines.

We compare meta-agent performance against two kinds of baselines as proxies to human performance: (1) Naive Agent—a minimal agent implementation that uses standard evaluation prompting without optimization, representing the lower bound of what a simple artifact can achieve. (2) Established human-engineered frameworks—specifically Terminus-2¹¹1https://www.harborframework.com/docs/agents/terminus-2 and OpenHands [19] for SWE-Bench and Terminal-Bench. These represent the "expert engineering" ceiling the meta-agent seeks to match or exceed.

5 Results

5.1 Evaluation Integrity Validation

Before reporting scores, we establish that our measurements reflect autonomous agent development capability rather than reward hacking. This rests on two pillars: a post-hoc auditing agent that reliably flags cheating attempts, and empirical evidence that its flags correspond to behaviors actually induced by optimization pressure on frontier scaffolds. The findings below underwrite the integrity markers () attached to every run in Section 5.2.

Auditor validation via induced red-teaming.

Explicit adversarial prompting—such as directly instructing the code agent to find and exploit vulnerabilities in the evaluation framework—triggered safety refusals from frontier models in almost every attempt. This high refusal rate rendered direct prompting unsuitable for robustly red-teaming the benchmark. Therefore, to reliably induce reward-hacking behavior, we employed an indirect protocol: evaluating the agent in a zero-resource configuration (i.e., without a valid API key for the artifact agent). This artificial constraint created extreme optimization pressure, incentivizing the agent to discover unintended avenues for reward. Notably, across eight independent trials, seven produced clear policy violations and one produced a valid artifact despite the missing credential. Our auditing agent agreed with a human annotator on all eight verdicts, substantiating the reliability of the integrity labels used throughout Section 5.2. Beyond this red-team setting, the auditor also surfaced emergent attacks during ordinary development runs. A detailed case study of autonomous label exfiltration by GPT-5.3-Codex is provided in Appendix B.3.1.

5.2 Meta-Agent Performance

媒体内容 · 前往原文查看

Model	Meta-AIME				Meta-GPQA				Meta-LiveCodeBench
	Run 1	Run 2	Run 3	Avg	Run 1	Run 2	Run 3	Avg	Run 1	Run 2	Run 3	Avg
Human Baseline	0.750 / -	0.750 / -	0.700 / -	0.733 0.029	0.585 / -	0.621 / -	0.586 / -	0.597 0.020	0.545 / -	0.554 / -	0.566 / -	0.555 0.011
Claude Code
Claude-Opus-4.6	0.767 /	0.683 / -	0.783 /	0.744 0.054	0.616 /	0.520 /	0.581 /	0.572 0.049	0.508 /	0.585 /	0.579 /	0.557 0.043
Claude-Sonnet-4.6	0.767 /	0.783 /	0.800 / -	0.783 0.017	0.565 / -	0.585 / -	0.000 / -	0.383 0.332	0.452 / -	0.576 /	0.310 /	0.446 0.133
MiniMax-M2.5	0.298 / -	0.394 / -	0.227 / -	0.306 0.084	0.353 / -	0.222 / -	0.515 / -	0.363 0.147	0.170 / -	0.291 /	0.319 / -	0.260 0.079
Kimi-K2.5	0.317 / -	0.700 / -	0.033 / -	0.350 0.335	0.222 / -	0.212 / -	0.338 / -	0.257 0.070	0.037 / -	0.003 / -	0.040 / -	0.027 0.021
GLM-5	0.247 / -	0.404 / -	0.414 / -	0.355 0.094	0.515 / -	0.566 / -	0.545 /	0.542 0.026	0.307 /	0.152 /	0.235 / -	0.231 0.078
Codex
GPT-5.3-Codex	0.250 / -	0.017 / -	0.383 / -	0.217 0.185	0.348 / -	0.217 / -	0.323 / -	0.296 0.070	0.322 / -	0.266 / -	0.211 / -	0.266 0.056
Gemini-cli
Gemini-3.1-Pro	0.733 / -	0.417 /	0.700 / -	0.617 0.174	0.566 /	0.556 / -	0.500 / -	0.541 0.036	0.489 /	0.084 /	0.328 /	0.300 0.204

媒体内容 · 前往原文查看

Model	Scaffold	Meta-SWE-Bench				Meta-Terminal-Bench
		Run 1	Run 2	Run 3	Avg	Run 1	Run 2	Run 3	Avg
Human Baseline	Terminus-2	0.616 / -	0.624 / -	0.672 / -	0.637 0.030	0.315 / -	0.315 / -	0.348 / -	0.326 0.019
Human Baseline	OpenHands	0.552 / -	0.544 / -	0.536 / -	0.544 0.008	0.225 / -	0.303 / -	0.326 / -	0.285 0.053
Claude-Opus-4.7	Claude Code	0.640 /	0.652 / -	0.536 / -	0.609 0.064	0.393 / -	0.360 / -	0.427 / -	0.393 0.034
Claude-Opus-4.6	Claude Code	0.600 /	0.512 /	0.216 /	0.443 0.201	0.236 / -	0.247 /	0.303 /	0.262 0.036
Claude-Sonnet-4.6	Claude Code	0.420 /	0.220 /	0.480 / -	0.373 0.136	0.292 / -	0.348 /	0.247 / -	0.296 0.051
MiniMax-M2.7	Claude Code	0.000 / -	0.004 / -	0.008 / -	0.004 0.004	0.101 / -	0.034 / -	0.000 / -	0.045 0.051
GLM-5.1	Claude Code	0.528 / -	0.444 /	0.456 / -	0.476 0.045	0.258 / -	0.270 / -	0.236 / -	0.255 0.017
DeepSeek-v4-Pro	Claude Code	0.400 / -	0.124 / -	0.444 / -	0.323 0.173	0.315 / -	0.348 / -	0.371 / -	0.345 0.028
GPT-5.4	Codex	0.168 / -	0.500 / -	0.068 / -	0.245 0.226	0.213 / -	0.191 / -	0.146 / -	0.183 0.034
GPT-5.3-Codex	Codex	0.060 / -	0.412 / -	0.408 / -	0.293 0.202	0.202 / -	0.135 / -	0.202 / -	0.180 0.039
Gemini-3.1-Pro	Gemini-cli	0.464 /	0.248 / -	0.468 /	0.393 0.126	0.303 / -	0.157 /	0.236 /	0.232 0.073

Tables 1 and 2 present the evaluation results across the reasoning and agentic domains, respectively. For the reasoning tasks, the API model for the developed artifact to use is Qwen3-8B deployed on a dedicated A100 vLLM backend, whereas Claude Haiku 4.5 is utilized for the SWE-Bench and Terminal-Bench evaluations. Our analysis yields three principal findings:

Finding 1: Meta-agents rarely match human scaffolds, and the few that do are dominated by proprietary frontier models.

Only of meta-agent configurations exceed the corresponding human baseline average, with of these driven by proprietary frontier models (Claude Sonnet/Opus) and only one open-weight configuration (DeepSeek-v4-Pro) crossing the bar. No meta-agent fully surpasses the baseline on GPQA or SWE-Bench, and open-weight models in particular fail to match human-built scaffolds in any reasoning domain, underscoring a substantial capability gap between closed- and open-source code agents in autonomous agent development.

Finding 2: High inter-run variance exposes the brittleness of autonomous design decisions.

We observe that of configurations exhibit a standard deviation greater than , compared to a maximum of among human baselines. Crucially, this extreme variance is not merely evaluation noise, but a fundamental bottleneck hindering current code agents from achieving reliable autonomous agent development. It demonstrates that while current models can occasionally synthesize highly effective agents, they lack the robustness to consistently navigate the open-ended design space.

Finding 3: High optimization pressure induces spontaneous reward hacking.

Our post-hoc auditor (detailed in Section 5.1) flagged five trials encompassing distinct exploit classes. Crucially, the dual-container isolation, split-level authorization, and proxy enforcement mechanisms successfully neutralized every exploit attempt; no flagged run artificially inflated its test score. Consequently, we retain these runs in the aggregated averages to document the agents’ adversarial intent without penalizing them for unsuccessful exploits.

5.3 Success and Failure Modes Analysis

To understand the macro-level dynamics that separate strong meta-agent runs from weak ones, we instrument all trials with six features parsed directly from the evaluation system log: total runtime, time-to-first eval call, number of eval calls, eval-call success rate, temporal centroid of eval calls (0 = all at start, 1 = all at end), and the mean inter-call interval. We then regress the final test reward on each feature after subtracting the per-domain mean.

Figure 3 highlights two dominant predictors of performance: mean inter-call interval and total runtime. In contrast, features that a naive view of iterative optimization might prioritize—such as the number of evaluation calls, success rate, time-to-first-eval, and temporal centroid—carry surprisingly little predictive signal.

These findings suggest a clear trend: successful meta-agents do not treat the evaluation endpoint as a high-frequency feedback signal. They think longer between calls, invest more total compute in artifact design, and probe the scorer sparingly. To better understand these behaviors, we further conduct a qualitative analysis of the generated artifacts, revealing their distinct design choices and behavioral patterns:

Successful reasoning artifacts converge on simple sampling pipelines.

Contrary to the complex tree-search or planner-worker decompositions prevalent in representative agentic reasoning literature, none of the top-performing reasoning artifacts utilize these structures. Instead, they consistently converge on pragmatic design choices: parallel sampling with majority voting, prompt diversification to mitigate vote collapse, the integration of code-execution, and adaptive time budgeting.

Top agentic artifacts favor minimal ReAct-style loops.

The best SWE-Bench and Terminal-Bench artifacts are tool-use loops over a small toolset. They share three key choices: API calls augmented with prompt caching to minimize latency, pre-search warming from issue symbols to populate context prior to the initial LLM invocation, and a singular verification nudge forcing the model to verify all requirements before terminating.

Underperforming agents trap themselves in design local optima.

Underperforming agents frequently suffer from under-exploration, prematurely converging on a minimal artifact and ceasing iterative development. Others get trapped in local optima, committing early to a fundamentally flawed or low-ceiling paradigm and subsequently wasting iterations resolving trivial plumbing errors within this constrained framework instead of pivoting to a more viable design.

Critical failures stem from inflexible resource management.

We observe a systemic lack of temporal awareness in meta-agents: they rarely monitor the remaining time budget proactively during the development process, frequently exhausting their allocated limits and suffering abrupt termination. This deficit also propagates to their generated artifacts. Several catastrophic failures (reward=0) originate from artifacts that fail to checkpoint partial answers. Consequently, when the evaluation harness issues a timeout signal mid-pass, all previously computed predictions are discarded, resulting in an empty submission to the agent runner.

5.4 Effort-Reward Trade-off

Building upon our analysis of development dynamics, we further evaluate meta-agent efficiency through estimated API cost (Figure 4a) and development time (Figure 4b). Our results show that Claude-Opus-4.7 anchors the Pareto frontier, achieving the highest overall performance at optimal efficiency. The evolution from Opus-4.6 to Opus-4.7 reveals that these capability gains stem from superior per-step decision-making rather than a mere increase in computational volume. Notably, by reducing completion time by 46% and requiring 23% fewer agent turns than Opus-4.6 on Terminal-Bench, Opus-4.7 corroborates official claims of accelerated autonomous problem-solving and demonstrates highly effective iterative reasoning.

6 Conclusion and Limitations

We introduce the Meta-Agent Challenge (MAC), an open-source evaluation framework shifting the paradigm from object-level execution to meta-level autonomous agent development. By challenging models to iteratively design and optimize task-specific agents within a sandboxed environment, MAC rigorously evaluates their capacity as system architects while preventing reward hacking. Across five diverse domains, our findings reveal that meta-agents rarely match human-engineered baselines, and the few that do are dominated by proprietary frontier models, while their design reliability remains brittle and optimization pressure can trigger emergent misaligned behaviors.

Limitations.

MAC is inherently time-intensive due to the ultra-long-horizon nature of simulating complete iterative development cycles. Furthermore, by repurposing existing object-level benchmarks (e.g., SWE-Bench, AIME), MAC inevitably inherits their underlying limitations (e.g., narrow task distribution), and the persistent risk of base-model pre-training data contamination.

References

• [1] Anthropic (2026-02) Responsible scaling policy, version 3.0. Technical report Anthropic. External Links: Link Cited by: §1.
• [2] Center for AI Safety, Scale AI, and HLE Contributors Consortium (2026) A benchmark of expert-level academic questions to assess AI capabilities. Nature 649, pp. 1139–1146. External Links: Document, 2501.14249, Link Cited by: Appendix C.
• [3] J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, et al. (2024) Mle-bench: evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095. Cited by: §2.
• [4] S. Hu, C. Lu, and J. Clune (2025) Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.
• [5] N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica (2025) LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix C.
• [6] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2023) Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: Appendix C, §2.
• [7] Kimi (2026) Kimi k2. 5: visual agentic intelligence. arXiv preprint arXiv:2602.02276. Cited by: §1.
• [8] T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, et al. (2025) Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503.14499. Cited by: §1.
• [9] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: §4.
• [10] Y. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn (2026) Meta-harness: end-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052. Cited by: §2.
• [11] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. (2026) Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. Cited by: Appendix C, §2.
• [12] A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian, et al. (2025) Alphaevolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. Cited by: §2.
• [13] J. Qiu, X. Qi, H. Wang, X. Juan, Y. Wang, Z. Zhao, J. Geng, J. Guo, P. Li, J. Shi, et al. (2025) Alita-g: self-evolving generative agent for agent generation. arXiv preprint arXiv:2510.23601. Cited by: §2.
• [14] B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko (2026) PostTrainBench: can llm agents automate llm post-training?. External Links: 2603.08640, Link Cited by: §2.
• [15] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2024) GPQA: a graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, External Links: Link Cited by: Appendix C.
• [16] J. Schmidhuber (2003) Gödel machines: self-referential universal problem solvers making provably optimal self-improvements. arXiv preprint cs/0309048. Cited by: §1.
• [17] Harbor Framework External Links: Link Cited by: Appendix C, §4.
• [18] W. Wang, X. Xu, X. Xu, et al. (2025) Let it flow: agentic crafting on rock and roll, building the rome model within an open agentic learning ecosystem. External Links: 2512.24873, Link Cited by: Appendix C.
• [19] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2024) Openhands: an open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741. Cited by: §4.
• [20] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig (2025) OpenHands: an open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1.
• [21] J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese (2025) Browsecomp: a simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. Cited by: §2.
• [22] H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, et al. (2024) Re-bench: evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114. Cited by: §1.
• [23] C. S. Xia, Z. Wang, Y. Yang, Y. Wei, and L. Zhang (2025) Live-swe-agent: can software engineering agents self-evolve on the fly?. arXiv preprint arXiv:2511.13646. Cited by: §2.
• [24] X. Yin, X. Wang, L. Pan, L. Lin, X. Wan, and W. Y. Wang (2025-07) Gödel agent: a self-referential agent framework for recursively self-improvement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 27890–27913. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §2.
• [25] G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan (2025) Memevolve: meta-evolution of agent memory systems. arXiv preprint arXiv:2512.18746. Cited by: §2.
• [26] J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune (2025) Darwin godel machine: open-ended evolution of self-improving agents. arXiv preprint arXiv:2505.22954. Cited by: §2.
• [27] H. Zhou, S. Guo, A. Liu, Z. Yu, Z. Gong, B. Zhao, Z. Chen, M. Zhang, Y. Chen, J. Li, R. Yang, Q. Liu, X. Yu, J. Zhou, N. Wang, C. Sun, and J. Wang (2026) Memento-skills: let agents design agents. External Links: 2603.18743, Link Cited by: §2.

Appendix A Prompt and Additional Experimental Configurations

A.1 Audit Agent

The auditing agent is instantiated via the Claude Code Agent SDK, with Claude Opus 4.6 as its backbone and full filesystem and shell tool access scoped to a single trial directory. It is invoked once per completed trial with the prompt below, and emits a structured JSON verdict (CLEAN, SUSPICIOUS, or CHEATING) together with supporting evidence.

The following prompt is used by the post-hoc auditing agent.

instance_id