AI-discovered harness tweaks halve agent token costs without retraining anything

Token cost is a first-order constraint for long-horizon agents, and the usual responses—faster attention kernels, quantization, cheaper models—all operate below the harness. SoL-Pi works at the harness layer instead, using an AI-driven search loop to discover mechanisms that reduce how many tokens an agent consumes without retraining anything.

The search is structured as a broad-to-deep funnel. A research agent inspects execution traces from a base harness, proposes candidate changes, and tests them across 535 executable environments—495 drawn from real GitHub issue–pull request pairs, 40 synthetic tasks with verifiable success criteria. Across roughly 150 proposed directions and more than 3,000 runs, four mechanisms survive the capability-constrained selection process: Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer.

Each addresses a distinct source of overhead. Action Fusion collapses a file mutation and its follow-up command into one tool call, eliminating a round trip. Online Context Compact checks at plan-step boundaries whether compacting the accumulated context would save more than the cache-rewrite cost. ObservationPack sends large tool outputs in full for the first two provider requests, then substitutes a stable handle and a short excerpt—the agent can page through the original on demand. Evidence-Preserving Reducer uses a lower-cost auxiliary model to compress build and test logs into verified receipts, falling back to the original if verification fails.

On the 51-task EdgeBench evaluation, the complete four-mechanism stack retains 93.7% of the base Pi harness's average score while cutting recorded token traffic by 49.0% and API cost by roughly a third. The paper prices this at $8.75–$13.50 in estimated hourly savings relative to native Codex and Claude Code harnesses. Transfer to Opus 5—a backend the harness was never optimized against—holds up: 94.3% of Pi's score, 44.7% less token traffic, 33.5% lower API cost, with no further search or adaptation.

The multi-agent swarm experiment is worth attention. Twenty SoL-Pi workers coordinated by a single GPT-5.6 Sol agent reduce API cost by 26.8% relative to a comparable Pi-worker swarm over a two-hour kernel-optimization run, while the Pi swarm misses the tightest performance threshold and SoL-Pi passes it. A more efficient harness compounds across workers.

The paper is candid about limits. Mechanisms trigger less often and less intensively under Opus 5, likely because the harness was developed entirely on GPT-5.6 Sol trajectories. The authors frame multi-backend training as future work. The search itself is expensive enough that controlled scaling-law comparisons are not yet feasible. And the generalization results, while promising, are described as preliminary—the paper does not claim they establish a scaling law or that the gains will compound indefinitely.

The longer-term framing—"pretraining the harness" by exposing it to many tasks and updating from trajectories, analogous to model pretraining—is speculative but coherent. If a cheaper harness lowers the cost of the auto-research loop that builds its successor, efficiency becomes a resource rather than just an outcome. That recursion is a hypothesis, not a demonstrated result, but it is the right question to be asking.