Turning a visual skill into text throws away up to 59% of its recoverable accuracy

Notes on Agent Skills Should Go Beyond Text: The Case for Visual Skills (arXiv:2606.01414) — Binxiao Xu, Ruichuan An, Bocheng Zou, Hang Hua · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

Most reusable agent skills today are pure text — instructions, reasoning traces, summarized trajectories. For GUI manipulation, layout work, counting, and other visual-centric tasks, the knowledge worth reusing is spatial: hit regions, alignment, running state. Prose alone underspecifies or discards it, a gap this paper calls the textual bottleneck. Its fix is Visual Skill, a representation that pairs declarative text logic with explicit visual support and a binding rule for when to invoke it. Visual support comes in three forms — static priors (wireframes or annotated templates that fix a spatial convention across similar screens), dynamic priors (in-situ visual traces — anchors, counting marks, visited-region overlays — that carry state between steps), and interleaved visual skills (ordered text steps bound to the exact screenshot, frame, or region that justifies each one). AutoVisualSkill, the accompanying authoring pipeline, builds these artifacts automatically from a task trajectory — packaging textual logic, visual assets, and a binding manifest into one skill directory instead of compressing everything into prose.

Two controlled tasks isolate these mechanisms, holding textual rules identical across conditions and varying only whether a visual prior is attached. On GUI grounding (ScreenSpot, ScreenSpot-v2, GroundUI-18K, run on Qwen3-VL-32B-Thinking), pooled Point-in-Box accuracy climbs from 86.4% with no skill to 88.1% with a text-only skill to 91.1% with a visual skill, with larger relative gains on Mean IoU. Dense object counting (CountBenchQA, on Gemini-2.5-Pro) tells a sharper story: the text-only skill scores 93.00% accuracy — below the 94.24% no-skill baseline — while a matched visual skill, where the model plots and keeps visible coordinate anchors on counted objects, reaches 97.12% and cuts mean absolute error from 0.161 to 0.054. By the paper's own normalized degradation metric, textualizing the counting skill discards 58.9% of the accuracy a visual prior would have recovered — roughly six times the loss on GUI grounding (8-17%), meaning state-tracking tasks lose far more to textualization than static-convention ones do.