Give a small model screenshots of its own skills and its GUI success rate more than doubles — 10.78% to 25.40%

Notes on MMSkills: Towards Multimodal Skills for General Visual Agents (arXiv:2605.13527) — Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu · May 2026

Note published July 24, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

MMSkills packages each reusable procedure as more than a script. Alongside a compact descriptor and text procedure, it attaches state cards — when to apply the skill, when not to, and which visual cues confirm progress or failure — plus multi-view keyframes captured at different crops and points in the task. An agentic trajectory-to-skill generator builds this library automatically: it clusters public, non-evaluation interaction trajectories by workflow, drafts a procedure per cluster, grounds each step in the screenshots where it actually happened, and audits the result before it enters the library. At inference time, branch loading keeps the main agent's context clean. Candidate state cards and keyframes get checked against the live screen in a disposable side branch, and only a distilled decision — what applies, what to do, what to avoid, how to verify — passes back to the agent doing the task.

Across OSWorld, macOSWorld, VAB-Minecraft, and Super Mario Bros, adding MMSkills lifted nearly every tested model, and where it improved, the smallest model typically gained the most. Qwen3-VL-8B-Instruct's OSWorld success rate rose from 10.78% to 25.40%; frontier-scale Gemini 3.1 Pro moved a smaller amount, from 44.08% to 50.11%. The gains show up in behavior, not just outcomes: on OSWorld, Qwen3-VL-235B invoked skills more often (37.50% to 65.28%), needed fewer steps per trajectory (about 15 down to about 10), and stopped repeating itself as much, with exact-repeat actions falling from 21.8% to 6.2%. A loading-strategy ablation explains why the branch step earns its keep — feeding the same screenshots and state text straight into the main context without filtering hurt performance, while distilling that evidence through the branch step first was the design that performed best.

Key numbers

Qwen3-VL-8B-Instruct, OSWorld success	10.78% → 25.40%
Gemini 3.1 Pro, OSWorld success	44.08% → 50.11%
Trajectory length, Qwen3-VL-235B on OSWorld	15.22 → 9.87 steps
Skill invocation rate, Qwen3-VL-235B on OSWorld	37.50% → 65.28%
Qwen3-VL-8B-Instruct, macOSWorld success (no gain)	6.29% → 6.29%

What this means for skill authors

The result worth sitting with isn't the headline lift, it's the ablation that explains it. Dumping more screenshots and state text straight into an agent's context — the instinct most people reach for first — made performance worse, not better. Only the filtered, distilled version of that evidence helped. That's a structural argument, not a prompting tip: a skill's value comes as much from what a runtime deliberately withholds as from what it hands over. Text-only procedures leave 'what does success look like on screen' implicit, and that's exactly where GUI and game agents stall.

For skill authors building anything that touches a visual environment, this is a case for treating verified state — screenshots tied to explicit when-to-use and when-not-to-use conditions, not just prose steps — as a first-class part of the package, generated and audited rather than hand-written once and left stale. It also argues against a 'more context is safer' default in any skill-loading design; a filtering step between stored evidence and live context is doing real work, and skipping it is a plausible way to quietly regress an otherwise-good skill.

It's not universal, though. The one flat result in the paper — Qwen3-VL-8B-Instruct showed zero movement on macOSWorld even with MMSkills loaded — is a caution against assuming a well-grounded skill package transfers across every model and domain pairing. Below some competence floor in a domain, a model may not be able to act on procedural guidance no matter how well it's packaged. That's consistent with why SkillFed treats a skill's fit to a given agent and task as something to verify empirically, not assume, before trusting it in a live run.

Skills related to this research

frontend-design Frontend-design guides creation of distinctive, working web interfaces that reject generic AI output. It enforces a context-gathering protocol to understand your audience and brand personality, then commits to a bold aesthetic direction—whether brutally minimal, maximalist, or any other intentional style—before implementing production-grade code with meticulous attention to typography, color, layout, and interaction.★ 271 memory-kit Memory Kit sets up a .cowork/ directory in your workspace to store rules, identity information, long-term notes, daily logs, and heartbeat templates. Use it when you need to establish persistent context and configuration for your work environment.★ 406 pi-finder-subagent Pi-finder-subagent scans your local repository to pinpoint relevant files and code snippets, helping you narrow context before coding. It outputs a cited shortlist with line ranges and saves a focused report to guide your next steps.★ 406 prd prd converts feature requests into structured Product Requirements Documents, delivering actionable outcomes and concrete decisions. It's designed for executable documentation tasks, not high-level strategy or planning discussions.★ 406

Related notes

References

Zhang, Shao, Li, Lin, Fu et al., "MMSkills: Towards Multimodal Skills for General Visual Agents", arXiv:2605.13527 (2026)
Xie et al., 2024 — "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments"
Yang et al., 2025 — "macOSWorld: A Multilingual Interactive Benchmark for GUI Agents"
Wang et al., 2023 — "Voyager: An Open-Ended Embodied Agent with Large Language Models"
Zheng et al., 2025 — "SkillWeaver: Web Agents Can Self-Improve by Discovering and Honing Skills"