Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses

The executable scaffold around an LLM—its prompts, tools, memory, state management, control logic—has always been treated as something humans design once and then freeze. Hierarchical Self-Improvement (HSI) asks what happens if the scaffold rewrites itself, and the procedure doing the rewriting also rewrites itself, while a frozen outer anchor prevents the recursion from becoming unbounded.

The architecture has three nested scopes, all running the same frozen model. The innermost scope executes tasks. The middle scope rewrites the task harness across iterations. The outermost scope rewrites the rewriting strategy itself. Crucially, the meta-evolver's own execution logic is loaded from an immutable template—that's the anchor that stops the recursion. The same DeepSeek-V4-Flash model operates at all three levels; no stronger external proposer is involved.

A deliberate design choice isolates what's actually being measured: reasoning is disabled during task execution and enabled only during rewriting. This prevents inference-time compute from masquerading as harness improvement—a confound that has plagued prior work in this space.

On BALROG, a benchmark of long-horizon text-based games, the gains are real and consistent on moderate-difficulty environments: BabyAI, Crafter, TextWorld, and MiniHack all improve over the initial handcrafted harness in raw % Progress. On BabaIsAI sub-suites evaluated with held-out test splits, evolved harnesses generalize to unseen tasks—near-perfect progress on BreakStop and GoTo. The trajectory analysis is instructive: early iterations expose missing abstractions (hidden reward signals, inventory state), middle iterations introduce structured algorithmic components like BFS pathfinding, and later iterations refine or prune. The meta-evolver's contribution is to codify successful local discoveries into reusable heuristics that shape future search.

The paper is equally clear about where the approach fails. NLE, the hardest environment, yields essentially no improvement. The explanation is principled rather than apologetic: harness evolution requires informative feedback signals to guide selection, and it cannot overcome a backbone that lacks the underlying capability to generate useful interaction signals. These are named explicitly as a feedback-fidelity bound and a backbone capability bound. The statistical learning theory from a companion paper provides formal grounding—PAC guarantees hold under self-modification only when the reachable hypothesis family has bounded complexity.

The honest framing matters. This isn't a claim that self-improving agents replace stronger models. It's a demonstration that task-specific, continuously evolvable harnesses are a viable axis for extracting additional performance from a fixed model—with clear empirical limits attached.