OpenSkill's verifier never sees the answer key, yet agrees with it 61% of the time -- and the skills it certifies beat closed-world baselines by 8.9 points

Notes on OpenSkill: Open-World Self-Evolution for LLM Agents (arXiv:2606.06741) — Zhiling Yan, D. Song, Hanrong Zhang, Weida Liang, Yuxuan Zhang, Yutong Dai, Lifang He, Philip S. Yu, Ran Xu, Xiang Li, Lichao Sun · June 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Most self-evolving agent frameworks quietly assume away the hardest case: deployment with nothing but a task prompt, no example skills, no successful past runs, no verifier to grade attempts. OpenSkill is built for exactly that gap -- what it calls open-world self-evolution -- via a three-stage pipeline. Stage 1 sends the agent out to query documentation, repositories, and the web for two separate things: knowledge for building the skill, and independently checkable verification anchors -- reference values, dataset statistics, documented output formats -- that never touch the benchmark's actual answer key. Stage 2 drafts one to four candidate skills and grades them with a virtual verifier, an isolated LLM session that writes a deterministic test suite grounded in those anchors rather than in ground truth, then runs up to three rounds of failure-diagnosis-and-rewrite until a skill clears its own self-built tests. Stage 3 freezes the winner and deploys it zero-shot to a target agent; the hidden ground-truth tests get touched only at that final step, never during construction.

Across SkillsBench (11 domains), SocialMaze, and ScienceWorld, run on two separate target agents (Opus 4.6 / Claude Code and GPT 5.2 / Codex), OpenSkill posts the best automated pass rate in every benchmark-agent pairing. On SkillsBench overall it hits 43.6% (Opus) and 42.1% (GPT), beating the strongest closed-world baseline by 8.9 and 8.8 points respectively and landing within one to three points of the human-authored skill ceiling. Skills built entirely on Opus 4.6 undergo zero-shot skill transfer -- deployed unedited onto four unrelated weaker models -- and still lift their scores 5.5 to 14.8 points. And the virtual verifier, despite grading skills without ever peeking at a single ground-truth test, agrees with the real pass/fail outcome 60.7% of the time and covers 88.9% of what the human-authored tests actually check.