Splitting agent memory into skills and experience gains up to 11 points of task success

Notes on XSkill: Continual Learning from Experience and Skills in Multimodal Agents (arXiv:2603.12056) — Guanyu Jiang, Zhaochen Su, Xiaoye Qu, Yi R. Fung · March 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

XSkill splits a multimodal agent's reusable memory into two separate structures instead of one. A skill library holds stable, task-level workflows and tool templates in Markdown. A companion experience bank holds short, disposable condition-action notes in JSON, each capped at a few dozen words. Both are built and retrieved from the agent's actual screenshots rather than text logs alone. A rollout summarization step reads each image alongside the trajectory and records which visual cue — an inverted photo, a washed-out region — triggered which action. A cross-rollout critique then contrasts successful and failed attempts on the same task to decide what gets added or rewritten in the experience bank. A periodic consolidation pass merges near-duplicate entries and prunes the weak ones, so the knowledge base doesn't bloat over time. At inference, a task decomposition retrieval step breaks a new query into sub-needs and retrieves matching skill sections and experiences for each. It rewrites them against the current images and folds the result into the system prompt — a reference the agent can deviate from, not a fixed script.

Across five benchmarks spanning visual tool use, multimodal web search, and mixed reasoning, and across four backbone models (Gemini-2.5-Pro, Gemini-3-Flash, GPT-5-mini, o4-mini), XSkill lifts Average@4 success rate by 2.58 to 6.71 points over a tool-only baseline. Against the strongest prior memory method the gap opens further: up to 11.13 points, on TIR-Bench with Gemini-3-Flash. Ablating either stream costs 3 to 4 points on VisualToolBench, and the two components fail in different ways. Pull the skill library and structural mistakes spike — syntax errors climb from 11.4% to 20.3% of runs, tool-name errors rise eightfold (2 to 16). Pull the experience bank instead and tool selection barely moves off the no-memory baseline (65.96% code-interpreter calls versus 66.63%), while adding experience alone pushes that figure to 74.49% and the full pipeline to 76.97%. Knowledge accumulated on one benchmark carries over as zero-shot transfer to a benchmark in the same domain — VisualToolBench knowledge applied to TIR-Bench, MMSearch-Plus knowledge applied to MMBrowseComp — holding a 2-to-3-point edge over baselines. GPT-5-mini and o4-mini even gain 2.58 to 4.16 points just by reusing a knowledge base that Gemini-3-Flash accumulated, without running their own accumulation phase at all.