Executable skills beat prompted skills 51% to 20.5% on web search

Notes on Harnessing LLM Agents with Skill Programs (arXiv:2605.17734) — Hongjun Liu, Yifei Ming, Shafiq Joty, Chen Zhao · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

HASP turns each stored agent skill into a small executable module called a Program Function (PF): a should_activate() check that watches the agent's current state and proposed next action, paired with an intervene() step that either rewrites the action outright or injects corrective context back into the reasoning trace. An external agent harness wraps the base policy (Qwen2.5-7B-Instruct throughout), retrieves candidate PFs at every step, and lets them fire before the action executes — no fine-tuning required to see a benefit. The same interface does double duty: each firing logs the original action, the repair, and the downstream effect, which becomes training signal for post-training via SFT, rejection sampling, or on-policy distillation, and recurring failures get mined into new candidate PFs that must clear executable validation plus a teacher's review before joining the shared skill library.

On web-search reasoning (HotpotQA, 2Wiki, MuSiQue), PFs firing at inference time alone lift average accuracy to 51.0% — well above 20.5% for the identical skills expressed as prompt text and 31.2% for a multi-loop ReAct baseline. Adding an auxiliary teacher to help pick which PF fires brings it to 56.2%, a 25-point margin over that ReAct baseline. Training on PF-corrected traces plus controlled library evolution (HASP-Evolve + RS) pushes the web-search average to 60.3% — a 30.4-point gain over Search-R1 — and lifts math reasoning to 45.4% and coding to 69.9% pass@1. Skipping the validation gates when growing the library is not neutral: it collapses web-search accuracy from 60.3% back down to 36.3%. A mechanism analysis finds 65.1% of PF firings rewrite the action directly rather than just inject text. Behavior-correcting skills — read more evidence before answering, say — mostly go silent after training: two of them stop firing on 100% of their previously-triggered cases and a third on roughly 30-37%, while question-dependent skills like decomposing a multi-hop query stay active, going silent on just 3-12% of cases: training absorbs the habit-forming corrections but keeps leaning on the ones that depend on new input.