Self-written APIs make web agents 32-40% better — weaker models gain even more from them

Notes on SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills (arXiv:2504.07079) — Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Z. Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, Yu Su · April 2025

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

SkillWeaver turns a web agent loose on a new site and runs a tight loop: the agent discovers candidate skills, executes them repeatedly for practice, then distills the successful attempts into small, callable API functions instead of raw click-by-click logs. Later rounds draw on that growing skill library, so the agent composes earlier APIs into more complex ones rather than relearning a site from scratch every time.

On the WebArena benchmark and a set of real websites, agents armed with their self-written APIs finish more tasks: a relative success-rate gain of 31.8% on WebArena and 39.8% on real sites over an agent with no API library. The sharper result is skill transfer — APIs distilled by a strong backbone model still work when handed to a materially weaker one, lifting that weaker agent's WebArena success by as much as 54.3%. Procedural knowledge captured as code outlives the model that discovered it.

Key numbers

WebArena success-rate gain	+31.8% relative
Real-website success-rate gain	+39.8% relative
Weak-agent gain from strong-agent APIs	up to +54.3% relative (WebArena)

What this means for skill authors

The mechanism here — a successful action sequence promoted into a named, documented function — is the same move a vetted skill catalog makes by hand, except the author is now the agent itself, not a person writing a workflow. That collapses authoring cost close to zero. But what counted as "this works" during practice was the agent's own self-assessment of its trajectories, not independent verification — a self-improving agent can calcify a blind spot into a reusable API just as easily as it can capture a genuine shortcut. That's exactly the failure mode that scanning a skill's actual behavior, not just its self-reported success rate, is meant to catch before it gets redistributed.

The transfer result is the one worth sitting with: procedural knowledge, once compiled into an API, held its value across a real capability gap between models. That's the load-bearing assumption behind any skill-sharing system — that a skill authored during one agent's practice sessions stays useful, and safe, when a different, often weaker, agent picks it up downstream. What the paper doesn't examine is the reverse case: a narrowly-tested or subtly broken API making that same trip. Provenance — self-authored versus human-vetted, tested against how many trajectories, on what site version — is metadata that should travel with a skill alongside its measured uplift, not get discarded once the number looks good.

Skills related to this research

Rtk Rtk helps you set up and troubleshoot Rust Token Killer, a CLI tool that rewrites shell commands into token-efficient versions for AI agents. Install via Homebrew, script, or Cargo, then initialize RTK for your specific agent—Claude Code, Codex, Gemini CLI, Cursor, Copilot, Windsurf, Cline, or OpenCode. Use direct `rtk` commands like `rtk git status` or `rtk read` when automatic hooks aren't available.★ 38 status Status displays which OMG mode is running and reports its current progress, including active phases, task completion, and agent states. Use it to verify what's executing, monitor advancement through workflows, or confirm state after resuming operations.★ 1,847

Related notes

References

Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, et al. "SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills." arXiv:2504.07079 (2025).
WebArena: a realistic, reproducible web environment used as the primary evaluation benchmark in this paper.