AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Harness engineering for LLM agents has been a craft problem: someone reads failure logs, rewrites prompts, tweaks tool configurations, and ships a new version. AutoSaddler treats that entire loop as an offline learning problem and automates it.

The core idea is that a harness—the prompts, tool interfaces, and runtime control logic wrapped around an LLM—can be treated as code subject to iterative, evidence-based revision. AutoSaddler runs the agent on a mini-batch of training tasks, diagnoses failures by actively inspecting execution traces and the harness codebase (not just reading a summary), generates a structured patch, verifies it on the same mini-batch, and then decides whether to commit it based on performance on a held-out development set. Lessons from each cycle are stored in a directed acyclic graph called EvoDAG, which the Evolution Agent consults to compose the next candidate harness—potentially cherry-picking components from multiple prior lineages rather than continuing linearly.

The performance numbers are consistent across three benchmarks: gains of 9.0, 9.6, and 10.0 percentage points over base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, respectively. Against the strongest automated baseline on each benchmark, the margins are 7.4, 4.4, and 6.7 points. On Terminal-Bench 2.0, AutoSaddler also edges past a manually expert-tuned harness by 2.5 points.

The ablations are where the paper earns its argument. Three design choices each carry measurable weight. Replacing deep trace-and-codebase diagnosis with a single LLM reflection call drops GAIA2 Pass@1 from 62.0 to 57.8. Removing the patch taxonomy and phased scheduling—which sequences capability patches before steering patches—drops it to 53.3. Removing the generalization-aware selection mechanism entirely collapses performance to 50.6, the largest single-component degradation. The regression-rate analysis explains why: without development-set filtering and EvoDAG-guided reflection, the regression rate trends upward at roughly 0.16 percentage points per iteration rather than downward.

The efficiency story is also concrete. AutoSaddler reaches 72.3% development accuracy on GAIA2 after 147 leveraged traces; Meta-Harness plateaus at 61.5% after consuming 1,400. AutoSaddler's optimizer-side cost is $14.56 per generated patch, but its selective evaluation strategy triggers far fewer expensive task-agent rollouts overall.

One honest limitation the paper acknowledges: the framework assumes access to a training set with gold answers and a clear pass/fail signal—an assumption that holds on research benchmarks but less cleanly in production. The paper flags unsupervised or weakly supervised variants as future work, and the current scope excludes stateful tasks and memory-dependent settings.

The analogy to mini-batch gradient training is drawn explicitly—diagnosis as gradient estimation, same-batch verification as checking the gradient direction, EvoDAG as optimizer momentum—and the paper is candid about where it breaks down: textual gradients are inferred and fallible in ways numerical gradients are not, which is precisely why the verification and reflection machinery exists.