Matching skills to scene context takes 3D-agent tool use from 39% to 78%

Notes on Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning (arXiv:2606.07436) — Haoyu Li, Zhengdong Hu, Jun Wang, Hehe Fan, Yi Yang · June 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

MLLM agents doing 3D spatial reasoning — object counting, distance estimation, route planning — call on external tools for detection, depth estimation, and 3D reconstruction. But they default to the same one or two tools no matter what the scene needs. Ask an "object-to-object distance" question and the agent answers from reconstruction evidence that encodes relative position, not the depth grounding the question actually requires. Skill-3D fixes the mismatch with memory instead of a bigger model. Every completed tool-use trajectory gets logged into a Scene Memory, successful trajectories from similar scenes distill into reusable "skills" — a trigger condition, required evidence, tool order, argument template — and failed trajectories don't get discarded; they attach to the nearest skill as lessons. Skills come in two flavors: static ones fixed as task-level priors, and dynamic ones that keep merging new successful variants and absorbing failure-driven fallback rules. The Skill Library and the Scene Memory keep refining each other as trajectories accumulate. At inference, the agent retrieves a compact slate of candidate skills by scene signature and evidence type, then a policy narrows that down to what it will actually use — before it touches a single tool.

The team distills this behavior into small open models — Qwen3-VL-4B/8B — through a two-stage post-training recipe. Supervised fine-tuning on skill-guided trajectories teaches the retrieve-select-execute pattern first; then Group Relative Policy Optimization (GRPO) applies a composite reward over answer correctness, output format, and tool efficiency. Across four closed-source MLLMs — GPT-4o, GPT-5.4, Gemini-2.5-Pro, Gemini-3-Flash — and four benchmarks, scene-aware skill retrieval nearly doubles effective tool usage (the share of tool calls that return valid evidence and actually get used in the final answer): from 39% to 78% on VSI-Bench, with comparable jumps on BLINK, CV-3D, and MMSI-Bench. Averaged across the four closed models, VSI-Bench accuracy climbs from 42.9 to 64.5 — a 50% relative gain over skipping tools entirely — and Gemini-3-Flash's MMSI-Bench score rises 67%. Ablations confirm retrieval is the load-bearing piece: removing it costs 5.8 points of VSI-Bench average, more than removing either skill type or the failure lessons on their own. It's cheaper, too — skill-guided routing cuts inference time to 20.8 seconds per query versus 35.1 seconds for the strongest prior baseline, because the agent stops defaulting to the slow generic reconstruction tool (about 21 seconds a call) once the scene calls for something faster and more targeted.