A weak model with a distilled skill beats its unaided teacher — at 1,000x lower inference cost

Notes on Evidence Over Plans: Online Trajectory Verification for Skill Distillation (arXiv:2605.09192) — Yang Zhou, Zihan Dong, Zhenting Wang, Can Jin, Shiyu Zhao, Bangwei Guo, Difei Gu, Linjun Zhang, Mu Zhou, Dimitris N. Metaxas · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Most agent-skill generation pipelines get the order backwards: a model plans a solution before it ever touches the environment, and that plan becomes the skill. That's the wrong sequence — a skill is only as trustworthy as the evidence behind it, and a pre-execution plan encodes untested assumptions, not verified facts. The Posterior Distillation Index (PDI) is the fix: a trajectory-level score, a z-scored and equally-weighted combination, that rewards execution grounding and penalizes plan copying and "memo ossification" — belief stagnation across repeated attempts. Generating the trajectories PDI scores runs through SPARK (Structured Pipelines for Autonomous Runnable tasKs and sKill generation): a teacher agent repeatedly attempts a task inside a Dockerized environment, keeps a running exploration memo, and distills the successful run into a SKILL.md file. PDI isn't only a post-hoc grade — a memo-based proxy of it also intervenes online, during exploration, before a weak skill ever gets written.

Across 86 runnable tasks from the SkillsBench suite, SPARK-generated skills beat human-written ones on most of seven student models: GPT-5.4-mini reaches a mean reward of 0.52 with a SPARK skill versus 0.47 with the human-written equivalent. More strikingly, cheap students catch up to expensive teachers — GPT-5.4-nano with a SPARK skill (0.41) outperforms an unaided Claude Opus 4.6 (0.37), at up to 1,000x lower inference cost than the teacher's own exploration, with student runs costing as little as $0.02 per task. PDI itself predicts that transfer: it correlates with student skill gains at Spearman ρ=+0.364 (p=4.7×10⁻⁷), and trajectories in the best quadrant — low plan-copying, high execution-grounding — gain +0.377 mean reward versus only +0.028 for the worst quadrant (high plan-copying, low grounding). Used as an online intervention signal rather than a post-hoc filter, PDI lifts pass rates substantially: DeepSeek-Chat goes from 9.1% to 42.4% on the PDI-rerun task set. The effect isn't limited to the original domain — on ALFWorld, a text-based household-task benchmark, PDI-refined skills raise success from 16.7% to 40.0%.