Chain-of-Experience for Continual LLM Improvement

Most test-time compute research asks how many parallel samples to generate. Chain-of-Experience asks a different question: what happens when a model keeps the full history of its own attempts and the feedback it received, and uses that accumulating record to improve on the same problem over successive iterations?

The answer, across eight models and six benchmarks spanning math, coding, and knowledge tasks, is consistently positive. With only self-generated feedback — no external verifier, no ground-truth signal — models gain an average of 5.6% over no-feedback baselines while actually reducing API cost by 19%. That cost reduction is the counterintuitive result worth dwelling on: iterating with feedback produces less verbose output per round than the sprawling generations that no-feedback approaches tend to emit, so the compute budget gets reallocated into useful cycles rather than token inflation.

The paper is careful to distinguish four feedback regimes on a spectrum from implicit to explicit: no feedback (pure reflection on prior attempts), model self-critique, execution signals from code interpreters, and binary correctness oracles. Correctness feedback is acknowledged as an upper bound that is often unavailable in practice. Self-feedback is the practically deployable version, and it does most of the work — on coding tasks it captures the majority of the executor-feedback gain (73.4% vs. 75.0%) while costing 13.4% fewer API calls.

Two findings complicate the optimistic picture. First, the gains correlate with base ability: the average Pearson correlation between zero-shot performance and improvement capacity is 0.50 across benchmarks, meaning weaker models extract less from the same feedback loop. Second, the BrowseComp-Plus results show that self-feedback actively hurts on knowledge tasks requiring information outside the model's training data — the model confidently iterates toward wrong answers it cannot correct without external retrieval.

The spurious-feedback experiments are genuinely interesting. Feeding models exclusively wrong signals degrades performance by an average of 7.6% on AIME 2025, but stronger models (GPT-5 mini) drop only 2.5% versus o4-mini's 12.8%. Paradoxically, constant "incorrect" feedback sometimes forces deeper re-evaluation, while constant "correct" feedback induces overconfidence. Selective majority voting over early valid attempts largely recovers performance even under adversarial signals.

Memory-compression approaches — Dynamic CheatSheet and SimpleMem applied within-task — consistently underperform plain self-feedback, which the authors attribute to aggressive summarization discarding critical intermediate reasoning. The implication is that the full trajectory, not a distilled cheat sheet, is what the model actually needs.

Most gains arrive within the first 20 iterations; extending to 50 adds only marginal improvement (roughly 2% on AIME 2025 after the first 20 rounds). That saturation curve matters for anyone thinking about deployment: short loops are sufficient, and the returns from prolonged iteration are not worth the cost.