UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Two problems plague GUI agents in practice: training data that drifts toward cheap, short tasks, and instructions that leave too much procedural detail unstated. UI-Mate attacks both simultaneously rather than treating them as separate research questions.

On the data side, the system runs a closed-loop pipeline that constructs executable environments, filters rollouts with a two-stage multimodal judge, and routes every task through a hierarchical capability tree. That tree is the key mechanism: instead of counting trajectories by application, it tracks coverage at the level of fine-grained operations and rebalances generation toward whatever is underrepresented. The paper reports that capability-aware sampling improves multi-application task performance by several percentage points over training without it, with gains concentrated precisely in the previously underrepresented capabilities. A separate finding is blunt about where difficulty actually lives: real office documents are roughly twice the size of synthesized ones, and trajectories over them run about 52% longer. Environment realism, not instruction complexity, is the binding constraint.

On the interaction side, DemoCUA converts a recorded execution—human or agent—into a subtask-level workflow rather than a replay script. The formalization is careful: the demonstration is a prior, not a target. The live screenshot retains veto power. Training deliberately omits intermediate actions from the workflow so the model cannot copy steps blindly; at inference, the full sequence is restored because the model has already learned to treat it as guidance. The result is that one demonstration raises strict success on the 33-task self-demo subset from 17.2% to 35.4% and progress from 67.9% to 81.1%. On a 30-task OSWorld subset, the average score jumps from 40.3% to 65.8%.

The new OSWorkerBench benchmark is worth noting separately. Its 100 tasks span 41 applications, with a median trajectory of 68 decision turns and 38 tasks exceeding 100 turns. Evaluators decompose completion into 1–13 checkpoints, enabling partial-progress measurement. The benchmark distinguishes self-demos (same task, strong-agent rollout) from variant-demos (human recordings of related but non-identical tasks). Systematic results on the 45-task variant-demo set are explicitly deferred to future work—a 10-task pilot showed promise but not yet stable enough for a main claim.

UI-Mate-27B scores 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, leading open-weight systems in both. The gap to frontier closed models on WindowsAgentArena is single digits. The 9B model outperforms EvoCUA-32B despite being less than a third the size, which the paper attributes to data coverage and quality rather than parameter count. One unresolved tension: incorporating historical reasoning into RL training accelerates entropy collapse and hurts exploration, even though the same reasoning traces improve inference substantially. That asymmetry has no clean solution here and is flagged as open work.