Retrieving from a 210-skill library mines diamonds 2.5x faster than writing code on the fly

Notes on See and Think: Embodied Agent in Virtual Environment (arXiv:2311.15209) — Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tianbo Ye, Jenq-Neng Hwang, Gaoang Wang · November 2023 · European Conference on Computer Vision

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

STEVE is an embodied agent for Minecraft built from three linked modules: a vision encoder (EfficientFormer) that tokenizes what the agent sees, a LLaMA-2-13B instruction core split into four roles (planner, critic, curriculum generator, describer), and a code-action module that never writes new code. Instead, it matches each planned step against a fixed library of 210 skill scripts spanning eight categories — collecting and crafting among them — via skill retrieval, using cosine similarity between the step description and each script's embedding. Training runs in two stages: offline supervised fine-tuning on the STEVE-21K dataset (600 first-person gameplay clips across six terrains, 20K Minecraft QA pairs), then online fine-tuning inside the MineDojo/Mineflayer simulator across 5,000 episodes, with GPT-4 standing in as the expert supervisor.

Retrieving a known skill beats generating one on the spot. STEVE finds ten diamonds in 14 search iterations against Voyager's 35 — a 2.5x speedup — and collects 67 diamonds per 100 iterations where AutoGPT manages 7. Climbing the tech tree from wood to stone to iron tools, STEVE is 1.5x, 1.4x, and 1.3x faster than Voyager at each tier, and it's the only method to reliably reach the diamond tier at all: 3 of 3 runs succeed (around 106 iterations) versus Voyager's 1 of 3 and AutoGPT's 0 of 3. On a Minecraft knowledge benchmark judged by GPT-4, Claude-2, and human raters, the 13B model edges past GPT-4 itself, 8.12 to 8.04 out of 10. Strip out vision entirely and the agent still crafts iron tools fine — but it struggles once it reaches the diamond tier, pinning vision perception as the ingredient that only pays off once the task gets hard enough to need it.