Letting computer-use agents explore GUIs first turns near-zero success into 25–37%

Notes on OSExpert: Computer-Use Agents Learning Professional Skills via Exploration (arXiv:2603.07978) — Jiateng Liu, Zhenhailong Wang, Rushi Wang, Bingxuan Li, Jeonghwan Kim, Aditi Tiwari, Pengfei Yu, Denghui Zhang, Heng Ji · March 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

General-purpose computer-use agents train on demonstrations from roughly 100 digital environments, then get thrown at whatever unfamiliar interface a real task requires — and the skills don't transfer. OSExpert's fix is to have the agent explore before it's ever asked to perform. A GUI-DFS (depth-first search) procedure pushes candidate UI targets onto a stack, resets the environment, replays the action sequence needed to reach each one, and coordinates three modules — planner, action executor, feedback classifier — to expand, terminate, or retry each branch. Any state the feedback module marks terminal becomes a verified unit-function skill with a short usage description; the agent then chains unit skills into composite tasks and folds those in too. A separate database of fine-grained action primitives — calling an external segmentation tool, extracting boundary coordinates, clicking a sequence of points along an object's contour — gets pulled in whenever the feedback module flags an error needing pixel-level precision, and a primitive only survives if it succeeds under verification. Once exploration wraps, a LoRA-tuned lightweight model trained on the resulting plan-skill pairs serves as a fast planner, producing a full plan in one forward pass instead of re-planning at every step, and a skill-boundary check lets the agent recognize tasks resembling past repeated failures and bail early instead of burning its inference-time-scaling budget.

To measure the gap this closes, the authors built OSExpert-Eval, a 113-task benchmark spanning six environments (GIMP, the three LibreOffice apps, Tableau, a custom MiniWord editor) designed to stress long-horizon composite workflows, unseen and creative UI layouts, and fine-grained manipulation — categories existing benchmarks like OSWorld barely touch. General baselines (OpenCUA-7B, OpenAI's Computer-Use-Preview, Qwen-3-VL-8B, CoAct-1, Agent-S3 with GPT-5) mostly collapse to 0–10% success on unseen UIs and fine-grained tasks; even the strongest baseline tops out near 10% on long-horizon composite tasks, taking 623–1231 seconds per task against 11–48 seconds for human experts. Agents running OSExpert's self-explored skill set land 25–37% success across every category, finish in 27–44 seconds, and close roughly 80% of the latency gap to humans. The ablations point the accuracy gains at environment-specific procedural knowledge for long-horizon and unseen-UI tasks and at the primitive database for fine-grained control; the latency win, though, comes mostly from the skill-boundary check, not the fast planner.