Verify a skill before you trust it — that's an 11% jump in success rate

Notes on Inducing Programmatic Skills for Agentic Tasks (arXiv:2504.06821) — Z. Wang, Apurva Gandhi, Graham Neubig, Daniel Fried · April 2025

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Agent skill induction (ASI) lets a web-browsing agent mine its own successful task runs for reusable Python functions, instead of jotting down prose notes. Once a task finishes, the agent drafts candidate skill functions — say, a search_product(name) that chains together clicks and form fills — then rewrites the original trajectory to call those functions as a prefix and lets itself finish the task from there. A skill only earns a permanent slot in the agent's action space once that run clears three checks against the live environment: an LLM evaluator confirms the task was actually solved, the run actually invoked the new skill, and the skill's calls actually changed something on the page, not just executed without error. The setup is tested on WebArena (812 tasks spanning shopping, forums, dev tooling, content management, and travel/maps), with Claude 3.5 Sonnet running the agent, the evaluator, and the induction step itself, against a static non-adaptive baseline and against AWM, a prior method that induces the same kind of workflow knowledge but keeps it as free-text notes in memory instead of executable code.

ASI reaches 40.4% success on WebArena — a 23.5% relative gain over the 32.7% static baseline and an 11.3% relative gain over AWM's 36.3% — and does it in fewer steps, cutting step counts 10.7-15.3%. An ablation on the shopping subset pulls the gain apart: re-inducing AWM's unverified text skills with execution-based verification, turning them into verified programs but still only keeping them in memory for reference rather than as callable actions, recovers 4.2 points of success rate by itself; moving those same verified programs out of memory and into the callable action space, as ASI does, adds a further 3.7 points. Neither verification nor callability alone gets you the full gain — you need both. The gap widens on longer, multi-step "scaled-up" tasks, like updating billing and shipping address together: success improves 20.7-38.9 points over baselines, and one worked example that takes AWM 27 steps takes ASI 4. Transplanted onto real sites (Target, Reddit, Google Maps) after training on WebArena's sandboxed equivalents, skills like product search reuse cleanly; skills tied to a specific UI, like a dropdown-based sort, quietly stop applying once the real site uses a different pattern, like a sidebar — though the agent mostly catches the mismatch and skips the stale skill instead of misfiring.