Harness-Zero: Harness Distillation via Agent-as-Harness

The core problem Harness-Zero addresses is straightforward: harness engineering improves agent performance, but those improvements stay trapped in the harness. Swap the harness out, and the gains evaporate. The paper's answer is to distill harness-induced behavior directly into model weights, so the deployed agent needs only a minimal scaffold.

The mechanism is called agent-as-harness. Rather than wrapping the student model in optimized code, a separate harnessing agent sits at the student's response boundary and reviews each proposed action before it executes. When a proposal is sound, it passes unchanged. When it isn't, the harnessing agent makes the smallest coherent correction expressible in the student's own action space. The student never sees the review; only the accepted response enters its trajectory. After collection, supervised fine-tuning on those reviewed rollouts bakes the behavior into the student's weights.

The numbers are worth examining carefully. On frontier models without any training, agent-as-harness averages 81.1% across six benchmark-model settings, against 78.1% for the code-as-harness baseline. That gap is modest. The more striking result comes from distillation: fine-tuning Qwen3.5-9B on reviewed trajectories raises its macro-average from 23.3% to 44.3%, and the distilled model under a bare minimal harness actually exceeds the base model running with the full evolved harness attached (41.7%). The distilled model also recovers 82.3% of the 28 harness-exclusive behavioral patterns on average.

The ablations are the most instructive part. Fine-tuning directly on strong-model trajectories leaves the student at its 12% base rate, even though that teacher succeeds on 52% of collection tasks. Giving the harnessing agent oracle answers raises collection success to 98.6% but yields only 15% after distillation. The harness-guided review, which has no access to answers and succeeds on only 59.4% of collection tasks, produces a 30% distilled model. Collection success does not predict distillation value; what matters is whether the supervision instills reusable procedural behavior the student can execute independently.

The method has real limits. It depends on a capable harnessing model: on weaker models the harnessing agent intervenes aggressively and harmfully, replacing 66% of steps on the weakest tested model while making things worse. Review latency is also substantial—USPTO trials take roughly 237 seconds on average versus about 100 seconds for the baseline. And deep domain knowledge transfers poorly: USPTO's chemistry-specific tooling and reaction priors survive only partially, with the distilled model reaching 30% against the harness-equipped baseline's 38%.

The broader implication is that harness development could become a scalable training-data source rather than a permanent deployment dependency—each specialized harness returning its gains to shared model weights instead of accumulating as fragmented external scaffolding.