One Rollout, 58.3% Success — Up to 61% Fewer Tokens Than Baseline Agents

Notes on PANDO: Efficient Multimodal AI Agents via Online Skill Distillation (arXiv:2605.24785) — Yubo Li, Yidi Miao, Yu Shen, Yuxin Liu · May 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill retrieval

Web agents that click through visual interfaces usually get more capable by spending more at inference: sampling multiple rollouts, adding a verifier pass, or running an offline discovery phase before the real task starts. Trace VisualWebArena trajectories and the waste has a shape: agents loop on repeat actions that burn steps without changing the page state, tool-discovery phases quietly cost roughly 100 steps per tool once a library passes 50 tools, and most baseline pipelines reuse under 11% of their prompt through caching. PANDO answers with a single-rollout design — no sampling, no pre-evaluation discovery budget — built around a structured Skill Library of pattern-triggered rules and parameterized routines, a reflector that checks whether a step actually moved the task forward, confidence-based demotion for skills that keep failing, hierarchical routing that reserves expensive reasoning for novel planning, and a prompt layout built to keep prompt cache utilization high.

On the full 910-task VisualWebArena benchmark, PANDO reaches 58.3% success, ahead of the SGV verifier baseline (54.0%) and a WALT reproduction (45.2%) — while spending 58% fewer tokens than SGV and 61% fewer than WALT. The intrinsic efficiency numbers move the same direction: a 9.1% action-repetition rate against 14.2% and 18.3%, and 72.4% cache utilization against 45.1% and 38.6%. A 300-task ablation shows where the accuracy actually comes from. Stacking rules, seed routines, the reflector, online distillation, skill-merging, and demotion carries success rate from a 38.6% backbone-only baseline to 57.3%. Layering hierarchical routing, visual compression, and cache-aware prompting on top adds only 1.7 more points of success (to 59.0%) but drops the token bill from 147K to 117K per task — those three components are a cost lever, not an accuracy lever. Across the full run, the Skill Library grows from 12 seed routines to 47 induced ones by task 910, with 32 surviving after 15 demotions and 11 merges.

Key numbers

Success rate, 910-task VisualWebArena	58.3% (vs. 54.0% SGV, 45.2% WALT reproduction)
Token cost per task vs. baselines	~115K tokens — 58% fewer than SGV, 61% fewer than WALT
Hidden tool-discovery cost (WALT-style pre-eval)	~100 steps per tool once library exceeds 50 tools
Prompt cache utilization	72.4% (PANDO) vs. 45.1% (SGV) / 38.6% (WALT)
Skill-learning core's accuracy contribution	38.6% → 57.3% SR in 300-task ablation

What this means for skill authors

The ablation is the real finding here, not the leaderboard number. Rules, routines, reflection, online distillation, merging, and demotion — the parts that actually teach the agent something — carry success rate from 38.6% to 57.3%. Routing, visual compression, and cache-aware prompting add barely more accuracy (1.7 points) but cut token cost by roughly a fifth, from 147K to 117K tokens. That's a separation worth copying: build the pieces that make a skill correct before building the pieces that make it cheap. A team that reaches for caching and model routing before its skill set is stable is optimizing a number that doesn't matter yet.

Confidence-based demotion is the detail worth stealing outright. Each skill in the library carries a running confidence score, and repeated failure pushes it onto a blacklist rather than letting a broken routine sit in the retrieval pool degrading every task that touches it. That's the same discipline any vetted skill catalog needs, ours included — a skill that passed review last month and breaks against a changed API or page layout today shouldn't stay trusted just because nobody's looked again. Vetting has to be a standing check, not a one-time gate.

Three intrinsic metrics matter more here than the headline success rate: action repetition rate, step overhead ratio, and cache utilization. They matter because a terminal success rate alone hides where the cost went — a WALT-style system paying roughly 100 steps per tool for discovery can post a respectable success rate while still being expensive per task in a way the headline number never shows. Anyone comparing agent-skill setups, or comparing skills against each other, should be asking for that kind of trajectory-level accounting, not just pass/fail.

Skills related to this research

Baoyu Image Gen Baoyu Image Gen creates images from text prompts using OpenAI, Google, or DashScope providers. Configure default settings, choose quality levels and aspect ratios, and optionally use reference images to guide generation.★ 331 Canghe Image Gen Canghe Image Gen creates images from text prompts across multiple AI providers including OpenAI, Google, DashScope, and Canghe. Configure your preferred provider, quality level, and aspect ratio, with support for reference images and batch generation when needed.★ 418 gsc-ai-performance Track how your brand appears in Google's AI-powered search features and AI Mode results. This skill helps you understand your current visibility baseline in these emerging search formats, so you can optimize your content strategy for AI-driven discovery and maintain competitive positioning as search behavior evolves.★ 641

Related notes

References

Li, Miao, Shen & Liu, PANDO: Efficient Multimodal AI Agents via Online Skill Distillation, arXiv:2605.24785 (2026)
Andrade et al., Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification (2026)