Diff a task run with the skill against one without it: 73.9% reward, zero labels

Notes on SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing (arXiv:2606.14239) — Hao Gao, Haoran Chen, Canghai Wang, Shasha Guo, Liang Pang, Zhaoyang Liu, Huawei Shen, Xueqi Cheng · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

SkillAudit evolves an agent skill with no labeled outcomes to train against. The core mechanism, paired trajectory auditing, runs the same task twice per iteration — once with the candidate skill injected, once without — and treats whatever diverges between those two runs as the entire training signal. Twelve evaluators, bundled under the name Process-Aligned Contrastive Evaluation (PACE), read that divergence along four axes: procedural adherence, artifact correctness, format consistency, and where the skill measurably helped. Each verdict cites a specific passage in the skill document, quoting it directly rather than gesturing at it. A separate, frozen structural verifier — compiled once from the task spec and never touched again — checks hard constraints like file existence and format compliance, and can veto or roll back an edit that makes things worse. From there, edits split into two modes: Refine strips noise from a skill that's already broadly working; Repair replaces passages that actively fight what the task needs.

Across 89 containerized tasks spanning 8 professional domains — software engineering, finance, cybersecurity, natural science, and others — skills evolved this way land a 73.9% average task reward, against 56.7% for the benchmark's static expert-written skill and 40.9% for no skill at all. The gain isn't uniform: SkillAudit beats the static skill in 7 of 8 domains, by as much as 38.5 points in software engineering, but ties it in finance and economics. There's a sharper asymmetry underneath that average — tasks where the starting skill already scored above 0.5 keep that result 92% of the time under evolution, but tasks starting below 0.5 only get lifted into passing territory 43% of the time. That gap traces to observability: skills built on executable, checkable knowledge such as APIs or formulas land around 79.9% reward, while skills carrying procedural or semantic guidance that leaves no artifact trace land around 69.2% reward. The audit loop can only act on what shows up as a difference it can actually check.