Chain-of-Experience：让 LLM 在测试时通过反馈迭代持续改进

1

UC Santa Cruz 2 Bytedance Seed

• equal contribution and work done at Bytedance Seed

Abstract

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models’ ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a con-tinual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels ( e.g. ,model and correctness signals) yields additional gains, and that CoE deliv-ers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

1 Introduction

Humans naturally and continuously learn from their experiences, each success or failure contributes to an evolving understanding that informs future decisions. Contemporary machine learning systems — modern large language models (LLMs) in particular (Achiam et al., 2023; Team et al., 2023; Grattafiori et al., 2024; Liu et al., 2024a; Yang et al., 2025) — behave quite differently: once trained, they are deployed in a fixed state, treating every inference as an isolated event and ignoring the rich feedback embedded in the problem solving process itself. In contrast, learning should be a continuous process, LLMs should be able to improve not only during training but also at test-time by iteratively interacting with their environment — processing feedback, engaging in reflection to constantly update their understanding, or shortly, improving from their own knowledge and experience (Silver and Sutton, 2025; Snell et al., 2025). Inspired by successes in reinforcement learning, search-based methods explore a vast range of solutions. Heuristic strategies (Yao et al., 2023; Hao et al., 2023), including majority voting (Wang et al., 2022; 2024a) aim to identify higher-quality answers leveraging verifiers from candidates parallelly generated by LMs. Nevertheless, the model experience from these methods is still temporary, which is usually consolidated into a single answer and then discarded, leaving models to restart each problem without accumulated insight. An alternative paradigm is self-refinement, where models are enabled to iteratively improve through self-critique and correction (Madaan et al., 2023; Shinn et al., 2023), internalizing feedback within a single context. Extensions such as self-debugging (Chen et al., 2023) and Reflexion Shinn et al. (2023) incorporate external signals like code execution signals, but they remain fragmented and shallow in the use of iterative experience. 1arXiv:2608.18027v1 [cs.CL] 18 Aug 2026 Preprint. Under review. ICL

Performance V4 Improving Capability Efficiency w/ Feedback w/o Feedback Dynamic CheatSheet Few-shot In-Context Learning Base OmniMath EvaLearn LiveCode Bench (V6) LiveBench (Code) OmniMath LiveBench (Code) OmniMath w/ feedback w/o feedback LiveBench (Code)

Figure 1: Summarized results on four benchmarks across math, code, and knowledge over four LLMs. Iterative problem solving under CoE provides three benefits: upper: by incor-porating feedback in CoE, the average performance across four LLMs outperforms other test-time augmentations; lower left: models are capable of achieving the better performance with lower API cost with feedback ; lower right: LLMs that perform better on the task display better improving capabilities through CoE with moderate to strong Pearson correlation. We present more explanations regarding baselines in Section 4.1. Building on these attempts, we conceptualize Chain-of-Experience (CoE), wherein models engage in iterative problem-solving with feedback, and investigate: “How can LLMs evolve from accumulated experience with interactions and feedback to improve during test time?” We use the CoE framework to systematically explore prolonged environment interactions across four feedback types: none, model feedback, executor feedback for code tasks, and correctness feedback for general tasks. To evaluate CoE-based algorithms, we prompt eight state-of-the-art models, including GPT-5, o3, Gemini-2.5 Pro, Claude-4.5 Sonnet, across math, coding, and knowledge domains. Extensive experiments over four feedback types—drawn from existing algorithms and our designed paradigm ( i.e. , none, executor, model, and correctness feedback)—show that CoE with feedback consistently yields notable improvements (Figure 1). In particular, simple CoE methods substantially outperform test-time scaling approaches that rely on experience from other tasks (Suzgun et al., 2025; Zhang et al., 2025b), achieving an average of 7-9% gain over these algorithms with just self feedback (average 62.9% to 71.0%). Moreover, models leveraging iterative experience and feedback reason more efficiently, delivering a 5.6% overall improvement with 19% lower API cost and higher accuracy per token across all tasks and models. Combining complementary feedback channels ( e.g. , model and correctness/executor signals) yields further improvements, while memory-based selection methods applied within task do not outperform full experience trails, suggesting that aggressive compression may discard critical intermediate reasoning. We further observe a clear positive correlation between learning gains from feedback and base ability, with an average Pearson correlation of +0.5 across five benchmarks, indicating that stronger models evolve more effectively from experience. Finally, our analyses reveal deeper behavioral insights: models remain robust under spurious or weak feedback, most gains occur early in the iterations, and distinct improvement trajectories emerge under different feedback types.

2 Related Work

Training-free Test-time Strategies. Large language models can reason without additional training via various test-time prompting strategies. Chain-of-Thought (CoT) elicits step-by-step reasoning and improves arithmetic, commonsense, and symbolic tasks (Wei et al., 2022), spawning a family of “Chain-of-X” methods (Yu et al., 2023; Li et al., 2023; Huang et al., 2023; Chia et al., 2023). Representative variants include contrastive CoT (Chia et al., 2Preprint. Under review. 2023), least-to-most prompting (Zhou et al., 2022), task-specialized forms such as Chain-of-Explanation, Chain-of-Note, and Chain-of-Knowledge (Huang et al., 2023; Yu et al., 2023; Li et al., 2023), and Tree-of-Thought (ToT), which explores multiple reasoning paths via search and outperforms CoT on planning tasks (Yao et al., 2023). With the emergence of large reasoning models such as OpenAI’s o series (Jaech et al., 2024) and DeepSeek R1 (Guo et al., 2025), verifier-based methods that select among parallel generations have regained attention, including step-level (Lightman et al., 2023; Wang et al., 2024b; Zhang et al., 2024) and output-level (Zheng et al., 2023a; Cai et al., 2024; Liu et al., 2025) verification. While effective as post-processing using external feedback (Liu et al., 2024b; Wang et al., 2025; Tu et al., 2025; Wang et al., 2024c), these methods lack the iterative loop for model evolving. Our CoE is also training-free, but differs by using feedback to drive iterative self-evolving during inference.

Learning from Experiences. Learning from experience underlies both human intelligence and AI systems. Reinforcement learning formalizes experience through policy gradients and actor–critic methods (Williams, 1992; Schulman et al., 2017), achieving success in games, robotics, and control (Silver et al., 2016), and more recently via post-training on online gen-erations to improve alignment and reasoning (Bai et al., 2022; Guan et al., 2024; Shao et al., 2024; Yu et al., 2025). Beyond training, experience can accumulate during inference. For cross-task experience, Dynamic CheatSheet (DC) (Suzgun et al., 2025), Agentic Context Engi-neering (ACE) (Zhang et al., 2025b), and related approaches (Zheng et al., 2023b; Wang et al., 2024d; Zhao et al., 2024) maintain persistent inference-time memories that distill reusable strategies, while agentic scaffolds enable collective experience sharing across agents (Tang et al., 2025; Chen et al., 2025a; Ouyang et al., 2025; Han et al., 2026). For same-task experience, Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), Self-Debug (Chen et al., 2023), S* (Li et al., 2025), Iteration-of-Thought (Radha et al., 2024), and ReVeal (Jin et al., 2025) iteratively refine outputs using self-feedback or execution signals, while recent pipelines further exploit offline model experience for agent improvement (Zhang et al., 2025a; Chen et al., 2025b). Most recent works (Xia et al., 2026; Tu et al., 2026; ?) also extend such paradigm to real-world usage. In contrast to prior work, we present a unified framework that treats a model’s entire solving history as experience and systematically studies diverse feedback signals to enable continual improvement at test time.

3 Model Improvement via CoE

In this section, we first provide general concepts of the iterative problem solving setup, termed Chain-of-Experience (CoE), followed by a detailed discussion of four diverse feedback types to enhance model experience. Finally, we provide explanations on how we scale up the iteration of experience to probe model learning performance at test-time.

3.1 Overview

In the traditional question-answering setting (Sutskever et al., 2014; Raffel et al., 2020; Roberts et al., 2020; Brown et al., 2020), when given a question Q, large language models (LLMs) will generate a plausible response A sampled from the conditional distribution

P(A | Q). To extend this paradigm into the era of experience (Silver and Sutton, 2025), we incorporate an environment feedback variable F to represent the observable consequence or evaluation of responses when grounded in an interactive environment. Formally, the feedback f is sampled from the conditional distribution f ∼ P′(F | Q, A), where P′ is modeled by an environment that can be instantiated as a coding execution environment, an internal world model of the agent, or even a real-world environment, which provides generative (Ouyang et al., 2022; Mahan et al., 2024) or environment feedback (Shao et al., 2024; Madaan et al., 2023) for the given sequence. In this paper, we investigate a paradigm that extends the single-turn formulation into a sequential decision process, where each response ai at step i depends on the full history and corresponding environmental feedback of prior attempts ; we refer to this setting as 3Preprint. Under review. Experience Action Feedback

Env 𝓦

Use 𝓦 to provide various feedback

Experience

Model 𝓜

Input

Action Feedback

Env 𝓔

Use 𝓔 to provide various feedback Use 𝓜 to learn from experience

Figure 2: An overview of our studies iterative improvement loop for LMs. The model M

learns by repeatedly interacting with the environment E (e.g. , model simulation or coding environment). In each cycle, M generates actions conditioned on the input and accumulated experience, then receives feedback from E to form a new experience. We investigate four specific feedback in this study: none, execution, model, and correctness feedback.

Chain-of-Experience (CoE). The generative process is defined:

at ∼ P(at | Q, e0, e1, . . . , et−1) (1) where ei is the ith experience consists of (ai, fi).

3.2 Feedback Spectrum

To fully characterize how different forms of experience shape the model’s iterative evolu-tion, we categorize feedback along a spectrum of richness — from completely implicit to strongly explicit signals. Each feedback type corresponds to a specific instantiation of the environment’s response, which in turn influences the next-step action via the CoE generative process. In general, the model can be updated according to

at ∼ P(at | Q, (a0, f0), . . . , (at−1, ft−1)) ,where the function fi is the feedback we employed at the i−th iteration. Below, we describe these four feedback types used in this study, together with their formal definitions.

No feedback. The environment provides no evaluation or signal, i.e. , fi = ∅. The experience reduces to ei = ( ai, ∅), the next action is sampled from

at ∼ P(at | Q, a0, a1, . . . , at−1),meaning any improvement must arise from reflection on prior attempts, without external guidance.

Execution feedback. For tasks grounded in executable or interactive environments ( e.g. ,coding tasks with interpreters or unit tests), feedback is generated by running the model’s response ai inside the environment E:

fi ∼ P′(F | Q, ai) = E (Q, ai),where fi may include execution traces, error messages, runtime logs, or test-case outcomes.

Model feedback. A (possibly identical) auxiliary language model Mfb acts as a judge or critic fi = Mfb (Q, ai), where fi may include textual critiques, preference scores, or struc-tured evaluations. This enables refinement even in the absence of an external environment, relying purely on linguistic or preference-based signals.

Correctness feedback. When a domain-specific verifier is available, the environment supplies binary correctness signals:

fi = 1{ai is correct } ∈ { 0, 1 }.Such oracle-like information provides explicit fine-grained evaluation of success and failure. Although this type of feedback is often unrealistic in real-world settings, where ground-truth verification is costly or unavailable, still, we include it as a high-signal reference setting to approximate an upper bound on the benefits of iterative refinement. 4Preprint. Under review.

4 Experiments

4.1 Experiment Setup Datasets. We focus on three different tasks: math, coding, and knowledge. More specific, we select two benchmarks for each task: AIME 2025 (Balunovi´ c et al., 2025), OmniMath (Gao et al., 2024), LiveCodebench (V6) (Jain et al., 2024), LiveBench (Code) (White et al., 2024), EvaLearn (Dou et al., 2025), and GPQA Diamond (Rein et al., 2024). Detailed descriptions are in Appendix C.

Baselines. As for baselines, we examine model skills in either (1) utilizing different levels of built-in reasoning or (2) leveraging experience from previously solved problems. For controlling reasoning depth, OpenAI and Claude models can be tuned to produce varying amounts of reasoning tokens. For methods that absorb experience from prior examples, we adopt few-shot in-context learning (ICL) (Brown et al., 2020) as a standard baseline. For a more sophisticated approach, we select Dynamic CheatSheet (Suzgun et al., 2025) and Agentic Context Engineering (ACE) (Zhang et al., 2025b), which maintain a continually updated external memory of reusable strategies. Although follow-up works (Ouyang et al., 2025; Cai et al., 2025) introduce finer-grained refinements in a similar processing loop, we use these two as the representative baseline. We select the most k ∈ [1, 5, 8, 12, 15, 20 ]

relevant solutions for ICL, DC, and ACE to form their context, we present baseline details in Appendix D.

Models. We focus on the latest language models with inherent reasoning abilities from various developers to probe their improving capabilities during test-time: GPT-5 (OpenAI, 2025a), GPT-5-mini (OpenAI, 2025a), o4-mini (OpenAI, 2025b), o3 (OpenAI, 2025b), o3-mini (OpenAI, 2025b), Gemini-2.5 Pro Comanici et al. (2025), Claude 4.5 Sonnet Anthropic (2025). We run all experiments for three times and report the mean and standard deviation statistics. We present detailed prompting configurations in Appendix A.

4.2 Scaling with Experience Findings 1 : Performance: The Chain-of-Experience setting with feedback boosts rea-soning LLMs on various tasks. LLMs equipped with feedback consistently outperform almost all baselines and settings. As shown in Figure 3, across six benchmarks, the Chain-of-Experience (CoE) paradigm—where models iteratively absorb and reuse prior feedback—delivers substantial performance gains (full results in Appendix H). With self feedback or executor/correctness feedback (as an upper bound), eight modern reasoning models achieve average improvements of 5.6% and 11.1% over their no-feedback counterparts, underscoring the value of explicit outcome-based signals for model refinement. For coding-centric tasks such as LiveBench (Code) and LiveCodeBench (V6), programmatic executor feedback derived from public test-case verification drives sharp accuracy gains of 8.6% on average (from 66.4% to 75.0%), while self-judgement feedback still provides a 7.0% lift. This indicates that models can internalize both fine-grained and abstract feedback cues into subsequent reasoning. On non-coding tasks ( e.g. , AIME 2025, OmniMath, GPQA Diamond), the same trend holds: correctness feedback establishes an upper bound, and self feedback—though noisier—continues to foster improvement ( e.g. , 75.1% > 67.1% > 62.5% w/o feedback). In comparison, although ICL, DC, and ACE remain widely used, none demonstrates reliable scaling under advanced reasoning models. Averaged across six benchmarks in Table 3, ICL, ACE, and DC achieve only 62.1%, 64.0%, and 62.7% respectively, all trailing a simple without baseline (66.8%). In contrast, incorporating explicit feedback yields consistent gains: results with model self feedback reaches 71.0% (+7-9% over ICL/ACE/DC), while the best feedback signal further improves performance to 79.3%. Overall, these results proves that feedback-driven CoE acts as a more general and effective test-time scaling framework, enabling LLMs to autonomously improve across domains over no-feedback inference and other baselines. 5Preprint. Under review. Self Feedback w/o Feedback

Dynamic CheatSheet Few Shot In-Context Learning Correctness/Executor Feedback Binary Executor Feedback High Reasoning Low Reasoning V6- with partial Claude and ations as x values LiveBench (Code) OmniMath LiveCode Bench (V6) EvaLearn AIME 2025 GPQA Diamond Agentic Context Engineering

Figure 3: Results of five state-of-the-art LLMs on six benchmarks using different generation techniques. Models under CoE with different levels of feedback (correctness/executor, self, and binary executor feedback) generally perform better than the baseline strategies (no feedback, DC, ICL, and the ones with different reasoning efforts). Results are averaged over 3 runs and we shade the standard deviation with a lighter color and plot bars showing model performance under different reasoning efforts. The full results are in Appendix H.

Findings 2 : Efficiency: LLMs with feedback strikes a balance between performance and API calling cost. As shown in Figure 4, incorporating feedback into CoE not only enhances performance but also improves efficiency — models with feedback tend to achieve higher accuracy at lower API costs (upper left of sub-figures). Across tasks, most LLMs using feedback-based variants (blue objects) consistently dominate the no feedback baselines (gray square), except Gemini 2.5 Pro. Full results are in Figure 12, Appendix H. For the two coding tasks, self feedback emerges as a cost-effective compromise: it captures the majority of the performance gain of executor ( i.e. , 73.4% vs. 75.0%) while incurring 13.4% fewer API calls ( e.g. , $70.7 vs. $81.6), and moreover, requires 20% less cost than the no-feedback counterpart. Similarly, on AIME 2025 and EvaLearn, self feedback provides more informative input with substantially lower API cost with 47.3% and 7.0% reductions across all eight LLMs, even compared with the no-feedback solution ( e.g. , $8.8 vs. $4.6 on AIME 25 and $325.3 vs. $302.4 on EvaLearn), while still yielding an average 4.4% and 6.9% accuracy improvement, respectively. One exception is Gemini 2.5 Pro, whose self-feedback produces more verbose output with smaller gains, suggesting it may be less effective at self-evaluation than other LLMs. Beyond API cost, we further analyze token-level efficiency in Table 5 (Appendix I). CoE with correctness/executor feedback achieves the best accuracy with total token counts comparable to other multi-round methods, yielding substantially higher return per token. For example, on AIME 2025, correctness feedback reaches 84.6% with 108K tokens, while DC uses only 11K tokens yet achieves a lower 74.7%, and no-feedback CoE consumes similar tokens (107K) but trails at 74.1%. This pattern holds consistently across benchmarks, confirming that CoE reallocates 6Preprint. Under review. LiveCode

Bench(V6) OmniMath GPQA Diamond V3 Self Feedback Correctness/Executor Feedback Binary Executor Feedback No Feedback

Figure 4: Total cost of each model over task completion vs. its best performance within 20 iterations. LLMs with feedback generally achieve higher scores with fewer costs (at the upper left), while iterative experience without feedback generally falls behind (at lower right). We provide full results over six benchmarks in Appendix H. 60 80

Base Capacity (%) 0.2 0.4 0.6 Learning Ability r = 0.33 AIME 2025 50 60 70 Base Capacity (%) 0.1 0.2 0.3 r = 0.24 OmniMath 60 70 80 Base Capacity (%) 0.0 0.2 0.4 r = 0.97 LiveBench (Code) 60 70 80 Base Capacity (%) 0.1 0.2 0.3 r = 0.83 LiveCodeBench (V6) 40 50 Base Capacity (%) 0.15 0.20 0.25 r = 0.37 EvaLearn 75 80 85 Base Capacity (%) 0.05 0.10 0.15 0.20 0.25 r = 0.28 GPQA-Diamond GPT-5 mini GPT-5 o4-mini o3 o3-mini Claude 4.5 Sonnet Gemini 2.5 Pro

Figure 5: Zero-shot performance of models (Base Capacity) and the learning gain show positive Pearson correlations (r), indicating that better-performing LLMs possess greater improving capability. Scores are averaged across 3 runs. compute into feedback iterations and generates less verbose outputs, rather than simply inflating prompt length.

Findings 3 : Improving Capability: LLMs that perform better on the task shows higher learning gain during test-time. We calculate the improving capability of a model M using ∆M = Smax −Sbase

1−Sbase , where Sbase

denotes the model’s initial zero-shot performance without feedback and Smax represents its peak accuracy achieved under our CoE setting with model self feedback. All numbers are averaged across three runs. In Figure 5, we observe a clear positive trend between base performance and improving capability across benchmarks. For instance, models on both coding tasks e.g. , LiveBench (Code) with r = 0.97) and LiveCodeBench (V6) ( r = 0.83) show strong correlations, indicating that models with stronger initial reasoning ability tend to learn more effectively from feedback. Although tasks like AIME 2025 ( r = 0.33) and OmniMath ( r = 0.24) show relatively weaker correlations, the overall trend remains consistent, with an average task-level Pearson correlation of 0.50. These findings suggest that learning from experience is an emergent property that scales with model capacity — stronger LLMs are inherently better at digesting feedback and improving through interactions. 7Preprint. Under review. Self Feedback (code) Executor 0/1 Feedback(code) Executor Feedback (code) Correctness Feedback (general) Self Feedback (general) No Feedback

Percentage 46.8% 25.6% 38.7% 59.0% 68.5% 29.4% 36.5% 27.8% 34.4% 20.4% 69.7% 19.7% 36.1% 33.5% 4.1% 6.0% 18.1% 4.2% 1.8% 2.5% 5.2% 12.2% Feedback Fidelity Self Reflection Specification Recall Random

Figure 6: Percentages of different reasons for LLMs’ improvement patterns from 6,630 incorrect to correct response pairs. We employ GPT-5 to conduct this automatic analysis.

5 Further Discussion and Conclusion

To further investigate LLMs’ learning capacity under CoE, we conduct ablations on the iterative problem-solving setup, including spurious feedback, improvement pattern analysis, dual feedback combinations, and principled experience selection. We additionally study model behavior under varying feedback strengths (Appendix F.3) and extended experience iterations (Appendix F.2).

Learning from Spurious Feedback. To investigate the robustness of LLMs under spurious feedback, we design an experiment where models receive exclusively incorrect or correct

feedback ( e.g. , always stating “the answer is incorrect” or vice versa). We report the best model performance over 20 iterations in Table 2. This setup evaluates whether models can recover from (or even benefit under) misleading feedback signals. We find that spurious feedback generally degrades performance by average 7.6% on AIME 2025 and 2.6% on GPQA-Diamond. Yet, stronger models such as GPT-5 mini exhibit greater robustness, with only minor drops of 2.5% and 0.6%, compared to o4-mini’s declines of 12.8% and 4.6%. To further enhance reliability, we introduce Selective Majority Voting (SelMV-n), which aggregates final answers via majority voting among the first n valid attempts. Interestingly, on GPQA-Diamond, SelMV with incorrect feedback surpasses model feedback by 0.9% (79.4% → 80.3%), while accuracies after SelMV improve by average 1.2% and 2.3% on AIME 2025 and GPQA-Diamond, respectively, underscoring the robustness of reasoning LLMs despite adversarial signals. Further results on spurious and different levels of feedback are in Appendix F.1 and F.3.

Analysis of Improvement Patterns. To better understand reasons for the improvement of LLMs through iterative feedback, we design the experiments to analyze the “why” behind each flip from incorrect to correct answer of LLMs. We collect 6,630 examples across all tasks and five models ( i.e. , GPT-5, GPT-5 mini, o4-mini, o3, o3-mini) and leverage the latest GPT-5 model to analyze the cause. To validate this automated analysis, we compute Cohen’s Kappa between GPT-5 and human ratings on 100 randomly sampled trajectories (25 per category), achieving 76.8% agreement, indicating substantial agreement (details in Appendix F.5). We define four factors behind the improvement of LLMs: Feedback Fidelity for improving from trail feedback, Self Reflection for referring to self-reflection, Specification Recall for correcting based on the question and/or format requirements, Random for model improving from other reasons. We present the detailed criteria of these factors and prompts in Appendix F.4 and K. In Figure 6, we observe: (1) 47.7% of all improvements are feedback-driven, confirming that LLMs meaningfully interpret and act on feedback signals. (2) In coding tasks, 30.0% of im-provements stem from specification recall, reflecting the format- and syntax-sensitive nature 8Preprint. Under review.

AIME 2025 LiveBench (Code) OmniMath Setting Acc Best R Acc Best R Acc Best R Dual (Model + Corr/Exec) 76.7 R19 81.2 R15 73.5 R17 Correctness / Executor 70.0 R13 78.1 R15 74.5 R17 Binary Executor ––71.9 R13 ––Model 60.0 R6 57.8 R17 50.5 R9 Model + DC 50.0 R8 51.6 R15 46.0 R10 Model + SimpleMem 56.7 R6 54.7 R17 49.5 R12

Table 1: Dual feedback and principled experience selection results using Claude 4.5 Sonnet. Dual feedback combines model feedback with correctness (math) or executor (code) signals. Memory-based methods (DC, SimpleMem) are applied within-task with no cross-task leak-age. Acc: best accuracy (%) over 20 iterations; Best R: iteration achieving best performance.

Feedback AIME 2025 GPQA Diamond GPT-5 mini o4 mini GPT-5 mini o4 mini Self 93.3 91.1 79.9 78.8 SelMV Self 91.1 88.9 80.4 79.5 All Correct 90.0 73.3 79.3 75.8 SelMV Correct 93.3 73.3 79.3 76.3 Incorrect 91.7 83.3 79.3 72.7 SelMV Incorrect 89.7 86.7 82.8 77.8

Table 2: The best performance over 20 iterations under constant correct ” or incorrect ” feedback (e.g. , “the answer is correct”). Selective majority voting (SelMV) helps LLMs maintain performance. Results are averaged over 3 runs with best scores emphasized .of these tasks. (3) Model-generated feedback elicits a higher feedback-related improvement proportion than other sources ( e.g. , 58.7% vs. 41.1%), suggesting self-generated feedback is more contextually aligned.

Dual Feedback CoE. To test whether multiple feedback channels are complementary, we combine model feedback with correctness (math) or executor feedback (coding) using Claude 4.5 Sonnet. As shown in Table 1, dual feedback shows clear complementarity: on AIME 2025, dual feedback reaches 76.7%, surpassing correctness-only (70.0%) and model-only (60.0%); on LiveBench (Code), dual feedback achieves 81.2% vs. 78.1% (executor) and 57.8% (model). On the more challenging OmniMath, gains are marginal (73.5% vs. 74.5% correctness-only), with correctness signals typically pushing best rounds later. These results suggest that feedback channels contribute complementary signals, though the marginal benefit depends on task difficulty. Full results are provided in Appendix J.

CoE with Principled Experience Selection. We further investigate whether memory-based selection and compression mechanisms add benefit beyond full experience trails by combining CoE with Dynamic CheatSheet (DC) (Suzgun et al., 2025) and SimpleMem (Liu et al., 2026) under the same within-task protocol (no cross-task leakage). As shown in Table 1, these methods do not outperform pure self-feedback: on AIME 2025, model feedback alone achieves 60.0% vs. 50.0% (+DC) and 56.7% (+SimpleMem); on LiveBench (Code), 57.8% vs. 51.6% (+DC) and 54.7% (+SimpleMem), suggesting that aggressive compression may discard critical intermediate reasoning.

Conclusion. We present a comprehensive analysis of Chain-of-Experience (CoE), showing that LLMs can improve during inference through iterative feedback and accumulated experience. Across math, coding, and knowledge tasks, methods in CoE consistently enhances performance and efficiency, demonstrating the effectiveness of feedback-driven test-time learning. Our analysis also reveals a positive correlation between model ability and improvement capacity, and shows that most gains emerge early, even under weak or spurious feedback. Finally, we analyze various model improvement patterns during this iterative process. 9Preprint. Under review.

6 Acknowledgement

The authors sincerely thank Deyao Zhu, Shu Zhong, and Guang Shi for providing valuable feedback and discussions on the experimental part and presentation of the paper.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. Anthropic. Introducing Claude Sonnet 4.5. https://www.anthropic.com/news/ claude-sonnet-4-5 , 2025. Accessed: 2025-11-17. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022. Mislav Balunovi´ c, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi´ c, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL

https://matharena.ai/ .Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems ,33:1877–1901, 2020. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 ,2024. Zhicheng Cai, Xinyuan Guo, Yu Pei, JiangTao Feng, Jiangjie Chen, Ya-Qin Zhang, Wei-Ying Ma, Mingxuan Wang, and Hao Zhou. Flex: Continuous agent evolution via forward learning from experience. arXiv preprint arXiv:2511.06449 , 2025. Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. Swe-exp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361 , 2025a. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 , 2023. Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, et al. Scaling agent learning via experience synthesis. arXiv preprint arXiv:2511.03773 , 2025b. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600 ,2025c. Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. Contrastive chain-of-thought prompting. arXiv preprint arXiv:2311.09277 , 2023. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 , 2025. 10 Preprint. Under review. Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, et al. Evalearn: Quantifying the learning capability and efficiency of llms via sequential problem solving. arXiv preprint arXiv:2506.02672 , 2025. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985 , 2024. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339 , 2024. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A Car-denas, Yuhui Xu, Ran Xu, Caiming Xiong, et al. Vlaa-gui: Knowing when to stop, recover, and search, a modular framework for gui automation. arXiv preprint arXiv:2604.21375 ,2026. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 , 2023. Fan Huang, Haewoon Kwak, and Jisun An. Chain of explanation: New prompting method to generate quality natural language explanation for implicit hate speech. In Companion proceedings of the ACM Web conference 2023 , pages 90–93, 2023. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.

arXiv preprint arXiv:2412.16720 , 2024. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar-mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami-nation free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 ,2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?

arXiv preprint arXiv:2310.06770 , 2023. Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. Reveal: Self-evolving code agents via iterative generation-verification. arXiv preprint arXiv:2506.11442 , 2025. Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. S*: Test time scaling for code generation. arXiv preprint arXiv:2502.14382 , 2025. Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. arXiv preprint arXiv:2305.13269 , 2023. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations , 2023. 11 Preprint. Under review. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024a. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.

arXiv preprint arXiv:2410.18451 , 2024b. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352 , 2025. Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao. Simplemem: Efficient lifelong memory for llm agents. arXiv preprint arXiv:2601.02553 , 2026. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36: 46534–46594, 2023. Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castri-cato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models.

arXiv preprint arXiv:2410.12832 , 2024. OpenAI. New embedding models and API updates. https://openai.com/index/ new-embedding-models-and-api-updates/ , 2024. Accessed: 2025-11-17. OpenAI. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/ , 2025a. Ac-cessed: 2025-11-17. OpenAI. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/ , 2025b. Accessed: 2025-11-17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022. Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140 , 2025. Santosh Kumar Radha, Yasamin Nouri Jelyani, Ara Ghukasyan, and Oktay Goktas. Iteration of thought: Leveraging inner dialogue for autonomous large language model reasoning.

arXiv preprint arXiv:2409.12618 , 2024. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1–67, 2020. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling , 2024. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910 , 2020. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathe-matical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024. 12 Preprint. Under review. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36:8634–8652, 2023. David Silver and Richard S Sutton. Welcome to the era of experience. Google AI , 1, 2025. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search.

nature , 529(7587):484–489, 2016. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations , 2025. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014. Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.07952 ,2025. Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, et al. Agent kb: Leveraging cross-domain experience for agentic problem solving. arXiv preprint arXiv:2507.06229 , 2025. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. Haoqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, and Cihang Xie. Vilbench: A suite for vision-language process reward modeling. arXiv preprint arXiv:2503.20271 , 2025. Haoqin Tu, Jianwen Chen, Zijun Wang, Siwei Han, Juncheng Wu, Hardy Chen, Haonian Ji, Kaiwen Xiong, Jiaqi Liu, Peng Xia, et al. Visualclaw: A real-time, personalized agent for the physical world. arXiv preprint arXiv:2606.16295 , 2026. Jianing Wang, Qiushi Sun, Xiang Li, and Ming Gao. Boosting language models reasoning with chain-of-knowledge prompting. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4958–4981, 2024a. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 9426–9439, 2024b. Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291 , 2025. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Pro-cessing Systems , 37:1474–1501, 2024c. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429 , 2024d. 13 Preprint. Under review. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsec-omp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516 , 2025. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314 , 2024. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein-forcement learning. Machine learning , 8(3):229–256, 1992. Peng Xia, Jianwen Chen, Xinyu Yang, Haoqin Tu, Jiaqi Liu, Kaiwen Xiong, Siwei Han, Shi Qiu, Haonian Ji, Yuyin Zhou, et al. Metaclaw: Just talk–an agent that meta-learns and evolves in the wild. arXiv preprint arXiv:2603.17187 , 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 , 2025. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.

Advances in neural information processing systems , 36:11809–11822, 2023. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau -bench: Abenchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045 , 2024. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 , 2025. Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210 , 2023. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems , 37:64735–64772, 2024. Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558 , 2025a. Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsid-har Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic con-text engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618 , 2025b. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19632–19642, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems , 36:46595–46623, 2023a. Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. arXiv preprint arXiv:2306.07863 , 2023b. 14 Preprint. Under review. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 ,2022. 15 Preprint. Under review.

Method AIME 2025 LiveCodeBench (V6) LiveBench (Code) OmniMath GPQA Diamond EvaLearn ICL 71.83% 62.50% 65.46% 53.12% 78.45% 40.99% ACE 71.98% 66.94% 69.38% 50.33% 76.58% 42.54% DC 73.33% 63.59% 68.58% 48.64% 79.56% 42.68% w/o Feedback 77.78% 72.57% 60.16% 65.17% 80.02% 44.91% Reasoning-high 69.05% 70.63% 55.46% 61.81% 76.21% 39.58% Reasoning-low 60.48% 61.03% 55.38% 50.60% 72.92% 29.34% Binary-Executor –72.90% 71.65% –––Self 82.22% 75.69% 69.94% 67.52% 81.03% 51.73% Correctness/Executor 89.05% 74.50% 75.78% 79.61% 99.52% 57.05%

Table 3: Average performance comparison (%) across different LLMs on different datasets. For baselines, ICL, ACE, DC stands for few-shot in-context learning, agentic context engi-neering, and dynamic cheatsheet, respectively.

A Detailed Experimental Setup

Different Reasoning Efforts. For OpenAI models, we employ the default ‘low’ and ‘high’ in the reasoning_effort parameter to twitch models’ reasoning level. For Claude 4.5 Sonnet, we disable the thinking mode and set the thinking budget to 10,000 as the low and high reasoning variants, respectively.

Decoding Parameters. For OpenAI models, we use the default decoding parameter with temperature set to 1.0. For Gemini, Claude series, we use a temperature of 0.2 for decoding.

B Averaged Results

We present the averaged best scores over 20 iterations of different methods in Table 3. From the table, we can clearly draw the conclusion that feedback helps LLMs perform better on all six benchmarks, while existing self-improving algorithms ( i.e. , ACE, DC) do not perform decently on this testing suite.

C Evaluated Benchmarks

We present detailed descriptions of our evaluated benchmarks below: • AIME 2025 (Balunovi´ c et al., 2025) is a challenging and universal math benchmark consists of 30 cases from AIME in 2025. • OmniMath (Gao et al., 2024) is a universal olympiad level mathematic benchmark consists of a total of 4,428 problems, and we sample 200 examples across all difficul-ties for evaluation. • LiveCodebench (V6) (Jain et al., 2024) is an evolving benchmark for challenging code generation. We select its latest version (V6) alone with a total of 175 samples. • LiveBench (Code) (White et al., 2024) is originated from a living benchmark spans across six aspects. We select the coding aspect, comprising 128 samples for evalua-tion. • EvaLearn (Dou et al., 2025) is the first benchmark that evaluate the experience learning abilities of language models, which includes total 648 examples. • GPQA Diamond (Rein et al., 2024) is a challenging multiple-choice question set in biology, chemistry, and physics, authored by PhD-level experts. It consists of 198 examples in total. 16 Preprint. Under review.

D Baselines

Built-in Reasoning : A native test-time scaling mechanism implemented in OpenAI models allows the reasoning effort to be adjusted between low and high. We treat this as a built-in and straightforward scaling baseline and present further details in Appendix A

Few-shot In-Context Learning (ICL). In few-shot ICL, each demonstration is a previously solved question–answer pair. As the model processes tasks sequentially, all past pairs are stored in an experience buffer. For a new question, we retrieve the k most similar past questions using embeddings from OpenAI’s text-embedding-3-large model (OpenAI, 2024), and include their original question–answer formats as demonstrations, followed by the new question. When fewer than k examples are available, all prior examples are used. We evaluate ICL with k ∈ [1, 5, 8, 12, 15, 20 ].

Dynamic CheatSheet (DC) (Suzgun et al., 2025). Dynamic CheatSheet (DC) is a test-time learning method that maintains an adaptive external memory of reusable strategies or code snippets distilled from prior solutions. As new problems are solved, DC summarizes high-level strategies from selected past tasks—using ground-truth answers for clean experience curation—and stores them as structured cheatsheets. Past tasks are retrieved using the same similarity-based retrieval as few-shot ICL; when fewer than k tasks exist, all available examples are used. Consistent with ICL, cheatsheets are synthesized from the most recent

k ∈ [1, 5, 8, 12, 15, 20 ] relevant solutions.

Agentic Context Engineering (ACE) (Zhang et al., 2025b). Agentic Context Engineering (ACE) is a context adaptation framework that treats prompts as evolving playbooks rather than static demonstrations. It incrementally distills reusable strategies and domain insights through a generate–reflect–curate process, producing localized context updates that preserve prior knowledge and avoid monolithic rewrites. Similar to few-shot ICL, ACE retrieves relevant past trajectories and integrates insights from the most recent k ∈ [1, 5, 8, 12, 15, 20 ]

trajectories into its evolving playbook.

E Benchmark Input and Answer Examples

We provide representative input and answer examples for three benchmarks below.

AIME 2025.

Solve the following AIME problem step by step and provide the final answer: 2x

• 2−x = 5.Compute 22x + 2−2x.Answer format: ANSWER: Answer: 21

GPQA Diamond.

Which of the following is not a property of an ideal gas under standard conditions? A) Molecules occupy negligible volume B) Collisions are perfectly elastic C) Internal energy depends only on temperature D) Pressure is independent of volume Answer: D

LiveCodeBench (V6).

Question: "Erase Leaves — Given a tree with N vertices, repeatedly remove leaf vertices. Find the minimum operations to delete vertex 1." Public: [{"input": "9\n1 2\n2 3\n...", "output": "5\n"}, ...] Private: [{"input": "2\n1 2\n", "output": "1\n"}, ...]

17 Preprint. Under review. We use exact match for AIME 2025 and GPQA-Diamond, LLM-as-a-Judge for Omni-Math (Gao et al., 2024), and a Python interpreter for coding tasks (LiveBench-Code and LiveCodeBench).

F Full Discussions

In this section, we present the full version of different discussions in

F.1 LLM with All Spurious “Correct” Feedback

In Figure 7, we further present model performance under two extreme conditions: receiving uniformly “correct” feedback ( e.g. , the answer is correct) and the SelMV-augmented results. We observe that although performance initially drops after exposure to such incorrect feed-back, which suggests temporary confusion in adapting to inconsistent supervision. The models quickly recover and even improve as they adapt to the underlying pattern. Interest-ingly, both GPT-5 mini and o4-mini exhibit larger gains when exposed to entirely incorrect feedback, as such feedback compels the models to re-evaluate their reasoning and verify their outputs. In contrast, consistently positive feedback tends to induce overconfidence, misleading the models into accepting their initial responses without critical reassessment. This observation suggests that, paradoxically, constructive noise (in the form of seemingly negative feedback) can sometimes stimulate deeper reasoning and enhance robustness in iterative test-time learning. AIME 25 GPQA-Diamond

V2 3 runs Self Feedback SelMV w/ Incorrect Feedback Incorrect Feedback SelMV w/ Correct Feedback Correct Feedback

Figure 7: Model performance using constant “incorrect” and “correct” feedback. By leverag-ing the selective majority voting, LLMs show decent performance when facing spurious feedback on math tasks.

F.2 Extended Rounds of Iterations

To further probe the learning capacity of LLMs, we extend the number of experience iterations from 20 to 50, as shown in Figure 8. Across both AIME 25 and OmniMath, we observe that most performance gains occur within the first 20 iterations, while later stages yield only marginal improvements ( e.g. , average 16.7% > 2.2% on AIME 25; 21.2% > 3.5% on OmniMath). This trend consistently holds across different models, suggesting that LLMs quickly internalize and consolidate the useful feedback signals in the early stages, after which learning saturates. These results highlight that the majority of test-time learning under CoE happens rapidly — shorter adaptation loops in our CoE are sufficient for most reasoning tasks, with less significant returns from prolonged experience accumulation. 18 Preprint. Under review. AIME 2025

Self Feedback Correctness Feedback OmniMath V2

Figure 8: Performance of GPT-5 mini, o4-mini, and o3-mini with extended iterations of experience to 50 on math tasks. We mark performance points within the first 20 iterations of experience in blue, and those from iterations 20 to 50 in green. Self Feedback Correctness Feedback External Model Feedback

V1 LiveBench (Code) Self Feedback Correctness Feedback External Model Feedback OmniMath AIME 2025 AIME 2025

Figure 9: Model performance of GPT-5 and GPT-5 mini with external model feedback on two math tasks.

F.3 Feedback Strength

For experience with model feedback, one intuitive exploration is to design CoE with different feedback providers. Specifically, we cluster GPT-5 and GPT-5 mini as a pair and allow each to serve as a feedback generator for the other on a mathematical and a coding task, as illustrated in Figure 9. Interestingly, external model feedback proves to be highly effective on these two benchmarks, where it even slightly surpasses correctness-based/executor feedback. For instance, GPT-5 mini achieves a peak accuracy of 94.4% with model feedback, compared to 93.0% with correctness feedback; similarly, GPT-5 reaches 93.3% vs. 92.2%, showing that high-quality model-generated judging signals can substitute explicit correctness supervision. On the more challenging OmniMath benchmark, GPT-5 continues to help its mini variant outperform its self-feedback baseline. However, because GPT-5 mini starts with only 62.5% zero-shot accuracy, it is unable to provide sufficiently reliable feedback to improve GPT-5, resulting in inferior performance relative to correctness feedback. Moreover, both GPT-5 and GPT-5 mini underperform their AIME 2025 results ( ≥60% vs. ≥80%), and correctness feedback remains clearly superior: GPT-5 gets 82.8% with correctness feedback but only 74.5% with external model feedback. Overall, these findings highlight a consistent trend: feedback quality correlates with the verifier’s base ability on the task, and model-generated feedback becomes competitive with correctness supervision only when the verifier is sufficiently strong, suggesting a practical threshold for deploying model-as-judge in iterative experience frameworks.

F.4 Analysis of Improvement Patterns

We present the detailed criteria that we used for classifying the 19 Preprint. Under review. • Feedback Fidelity: Improvements directly driven by external feedback, where the model explicitly incorporates provided guidance or corrections into its next response. • Self Reflection: Improvements arising from the model’s own reasoning, identifying and correcting errors with little reliance on external feedback. • Specification Recall: Adjustments motivated by task instructions or formatting requirements, as the model re-aligns with the original question or output schema. • Random: Changes with no identifiable cause, typically minor rewording or stylistic variations unrelated to feedback or specification.

F.5 Human–GPT Judge Agreement

To validate the GPT-5-based automatic improvement analysis, we randomly sample 100 incorrect-to-correct trajectory pairs (25 per category: Feedback Fidelity, Self Reflection, Specification Recall, Random) from the full set of 6,630 examples. Two human annotators independently classify each pair into the four categories using the same criteria as the GPT-5 judge (Appendix K). We then compute Cohen’s Kappa between the GPT-5 labels and the majority human label.

Category Agreement (%) Cohen’s κ Feedback Fidelity 84.0 0.81 Self Reflection 72.0 0.71 Specification Recall 80.0 0.78 Random 68.0 0.63 Overall 76.0 0.768

Table 4: Cohen’s Kappa agreement between GPT-5 judge and human annotators across four improvement categories on 100 sampled trajectories. The overall κ of 0.768 indicates substantial agreement ( ?). The overall κ of 0.768 falls in the “substantial agreement” range, confirming that GPT-5 is a reliable proxy for human attribution in this task. Disagreements concentrate in the Random and Self Reflection categories, where the distinction between stochastic drift and genuine self-correction can be ambiguous even for human raters.

G BrowseComp-Plus

We also report model performance on BrowseComp-Plus Chen et al. (2025c). It is a bench-mark to evaluate deep research systems, isolating the effect of the retriever with a local database. It is sourced from the BrowseComp Wei et al. (2025) and we sample 200 examples to accelerate the evaluation. In Figure 10, we observe that unlike coding and math tasks, BrowseComp-Plus requires knowledge beyond the scope of the models’ training data. Consequently, for most models, incorporating self feedback leads to a performance decline compared to the no-feedback setting, highlighting the limitation of relying solely on self feedback in out-of-distribution knowledge scenarios.

H Full Results of Performance and Efficiency

We present full results of model performance (Figure 3) and API costs (Figure 12) regarding eight reasoning LLMs over six benchmarks. 20 Preprint. Under review. Self Feedback w/o Feedback Dynamic CheatSheet In-Context Learning Correctness/Executor Feedback

Figure 10: On BrowseComp-Plus, self-feedback models fall behind as the task requires external search-based knowledge. V6

LiveBench (Code) OmniMath LiveCode Bench (V6) EvaLearn AIME 2025 GPQA Diamond Self Feedback w/o Feedback Dynamic CheatSheet In-Context Learning Correctness/Executor Feedback Binary Executor Feedback High Reasoning Low Reasoning Agentic Context

Figure 11: Full results of eight state-of-the-art LLMs on six benchmarks incorporating different generation techniques. Model accuracies with different levels of feedback (cor-rectness/executor, self, and binary executor feedback) generally perform better than the baseline strategies.

I Token Analysis

We report aggregated token counts alongside accuracy for representative methods in Table 5. CoE with feedback achieves higher accuracy at comparable token budgets to other multi-round methods, demonstrating that feedback-driven iterations yield higher return per token rather than simply inflating prompt length.

J Dual Feedback and Principled Selection

We present full results of dual feedback CoE and principled experience selection experiments using Claude 4.5 Sonnet. In the dual feedback setting, model feedback is combined with correctness feedback (math tasks) or executor feedback (coding tasks) within each iteration. For principled selection, we integrate Dynamic CheatSheet (DC) (Suzgun et al., 2025) and SimpleMem (Liu et al., 2026) within the same task (no cross-task retrieval) to test whether memory-based compression adds benefit beyond full experience trails. Dual feedback demonstrates clear complementarity on AIME 2025 and LiveBench (Code), where combining two feedback channels surpasses either channel alone. On the more challenging OmniMath, correctness feedback alone matches dual feedback, suggesting that 21 Preprint. Under review. LiveBench

(Code) AIME 2025 OmniMath EvaLearn GPQA Diamond V2 Self Feedback Correctness/Executor Feedback Binary Executor Feedback No Feedback LiveCode Bench(V6)

Figure 12: Full results of total API cost (in dollar) vs. best model performance over 20 iterations. LLMs with detailed feedback ( i.e. , self feedback) achieves decent results with fewer costs, while CoE without feedback generally falls behind (at lower right).

Dataset Method Tokens Acc (%)

AIME 2025 CEF 108,734 84.6

SF 108,231 83.8 NF 106,825 74.1 DC 11,233 74.7 OmniMath CEF 176,412 74.2

SF 175,806 72.1 NF 173,944 66.8 DC 16,904 63.9 LiveCodeBench (V6) CEF 224,118 72.6

SF 223,441 71.2 NF 221,550 68.0 DC 20,771 66.4

Table 5: Token complexity vs. accuracy across methods. CEF: Correctness/Executor Feed-back; SF: Self Feedback; NF: No Feedback; DC: Dynamic CheatSheet. Token counts are aggregated across all iterations. when the primary signal is already strong, the additional model feedback provides marginal benefit. Memory-based selection methods (DC, SimpleMem) consistently underperform pure model feedback, indicating that aggressive summarization or retrieval may discard critical intermediate reasoning steps that full experience trails preserve. 22 Preprint. Under review.

K Prompt for Improvement Pattern Analysis

DUAL-AXIS LIFT/CHANGE ATTRIBUTION JUDGE You are a dual-axis attribution judge. Your job is to identify both why the model changed and what specifically changed between two consecutive attempts on the same problem. Do not decide if the solution is correct overall — correctness labels are provided. Instead, attribute the observed change along two orthogonal dimensions: * Change Driver (WHY) – the motivation or trigger behind the change. * Change Manifestation (WHAT) – the concrete locus or type of modification made. —I. Change Driver (WHY the change occurred) These categories capture the source or motivation of the update in the second attempt. 1. Feedback Fidelity – The model directly uses the provided feedback to modify its output. Signals: Edits match failing test or critique location; added clause mirrors feedback. 2. Self-Reflection / Internal Reasoning – The model self-identifies an error or im-provement without (using) explicit feedback or tests a different approach to see if it performs better. Signals: “I realized. . . ,” “previously I miscalculated. . . ,” or internally consistent reformulation not prompted by feedback or when there’s no feedback. 3. Specification Recall / Compliance Awareness – The model remembers or re-aligns with task instructions or formatting requirements. Signals: Adds “FINAL ANSWER:” wrapper, adheres to requested schema, restores omitted step explicitly mentioned in prompt. 4. Random / Drift / Unknown Driver – The motivation cannot be inferred; the change appears stochastic or stylistic. Signals: Superficial rewording, minor ordering changes, no logical link to feedback or instruction. —II. Change Manifestation (WHAT changed) These categories describe the technical or structural form of the change between attempts. A. Structural Plan / Algorithm Revision – A new high-level approach or reformula-tion (e.g., brute force → DP, heuristic → formula). Signals: Rewritten main structure, new helper functions, change in complexity or data representation.* B. Local Step Soundness & Invariant Fix – Correction of a local logic, variable, arithmetic, or invariant violation while keeping the overall plan. Signals: Fixed off-by-one in loop, corrected variable update, repaired algebraic derivation.* C. Edge / Boundary Condition Handling – Added or fixed guard for extreme/special cases (e.g., empty, zero, overflow, tie). Signals: ‘if n == 0‘, ‘<=‘ ‘<‘, added epsilon, handled ‘len == 1‘.* D. Output / Format Compliance – Adjusted presentation or output schema without changing algorithmic content. Signals: Added “FINAL ANSWER:,” fixed JSON/CSV layout, printing only required token.* E. Other / Ambiguous Change – Cannot clearly assign to A–E or insufficient evidence. Signals: Stylistic edits, reordering, or unrelated cleanup.* — ... ... VII. Constraints * Do not recompute correctness or logic. * Focus on identifying the causal link (WHY) and the technical locus (WHAT) of the change. * Be concise: cite only minimal evidence sufficient to justify your judgment. * If code is long, highlight the 1–2 lines most diagnostic of change. * If feedback is visible but not followed, set ‘"visible": true, "operationalized": false‘. * Always describe the learning pattern observed in Attempt_t+1.

L Limitations

Our evaluation focuses primarily on math, knowledge, and coding benchmarks. While these domains offer controlled settings to probe iterative improvement, there are interaction-23 Preprint. Under review. intensive scenarios where experience unfolds over long horizons Jimenez et al. (2023); Yao et al. (2024), to which the CoE paradigm should naturally extend. In addition, we do not update model parameters in this study. This choice isolates the Chain-of-Experience mechanism as a test-time paradigm, but it also means that the observed improvements arise from contextual reuse of experience rather than true learning; incorporating weight updates to internalize experience remains an important next step toward training models with persistent the “learning-from-experience” abilities.

M Declaration of AI Tool Usage

During the preparation of this manuscript, we used OpenAI’s GPT-5 model for minor language refinement and smoothing of the writing. The AI tool was not used for generating original content, conducting data analysis, or formulating core scientific ideas. All con-ceptual development, experimentation, and interpretation were conducted independently without reliance on AI tools. 24