skillfed

Retrieving from a 210-skill library mines diamonds 2.5x faster than writing code on the fly

Notes on See and Think: Embodied Agent in Virtual Environment (arXiv:2311.15209) — Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tianbo Ye, Jenq-Neng Hwang, Gaoang Wang · November 2023 · European Conference on Computer Vision

Note published · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill retrieval

STEVE is an embodied agent for Minecraft built from three linked modules: a vision encoder (EfficientFormer) that tokenizes what the agent sees, a LLaMA-2-13B instruction core split into four roles (planner, critic, curriculum generator, describer), and a code-action module that never writes new code. Instead, it matches each planned step against a fixed library of 210 skill scripts spanning eight categories — collecting and crafting among them — via skill retrieval, using cosine similarity between the step description and each script's embedding. Training runs in two stages: offline supervised fine-tuning on the STEVE-21K dataset (600 first-person gameplay clips across six terrains, 20K Minecraft QA pairs), then online fine-tuning inside the MineDojo/Mineflayer simulator across 5,000 episodes, with GPT-4 standing in as the expert supervisor.

Retrieving a known skill beats generating one on the spot. STEVE finds ten diamonds in 14 search iterations against Voyager's 35 — a 2.5x speedup — and collects 67 diamonds per 100 iterations where AutoGPT manages 7. Climbing the tech tree from wood to stone to iron tools, STEVE is 1.5x, 1.4x, and 1.3x faster than Voyager at each tier, and it's the only method to reliably reach the diamond tier at all: 3 of 3 runs succeed (around 106 iterations) versus Voyager's 1 of 3 and AutoGPT's 0 of 3. On a Minecraft knowledge benchmark judged by GPT-4, Claude-2, and human raters, the 13B model edges past GPT-4 itself, 8.12 to 8.04 out of 10. Strip out vision entirely and the agent still crafts iron tools fine — but it struggles once it reaches the diamond tier, pinning vision perception as the ingredient that only pays off once the task gets hard enough to need it.

Key numbers

Diamond search speed14 iterations vs Voyager's 35 (2.5x)
Diamond-tier success rate3/3 runs vs Voyager 1/3, AutoGPT 0/3
Skill library size210 scripts across 8 categories
Minecraft QA score8.12/10 (STEVE-13B) vs 8.04 (GPT-4)
No-vision ablationreaches iron tier, struggles at diamond tier

Skills related to this research

wiki-retrieve wiki-retrieve upgrades vault search from page-level to chunk-level granularity, combining contextual prefixes, BM25 sparse retrieval, and dense reranking to reduce retrieval failures. Setup is opt-in with built-in privacy gates: run fully on-machine with synthetic prefixes, or opt into Anthropic API or claude CLI tiers for better context generation.★ 10,007 pnpm This skill enables agents to handle Node.js package installation and dependency management through pnpm, a performant alternative to traditional package managers. Leverage pnpm's speed and storage efficiency to streamline your project's dependency lifecycle.★ 5,665 Context7 Context7 fetches the latest library documentation to give you accurate, up-to-date API information and code examples. It eliminates hallucinations from outdated training data by retrieving version-specific details when you ask about library usage, request examples, or explicitly invoke it.★ 3 ai-bot-log-audit Track and interpret how AI agents interact with your infrastructure by examining server logs for bot activity patterns. Gain actionable insights into crawl frequency, request timing, and resource usage to refine your content strategy and improve bot-to-human traffic balance. Essential for teams managing high-volume AI agent access.★ 140
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability to interact with… Zheng et al. · similarity 0.89 Improving Agent Interactions in Virtual Environments with Language Models Enhancing AI systems with efficient communication skills for effective human assistance necessitates proactive initiatives from the system side to discern… Zhang et al. · similarity 0.88 Odyssey : Empowering Minecraft Agents with Open-World Skills Recent studies have delved into constructing generalist agents for open-world environments like Minecraft. Despite the encouraging results, existing efforts… Liu et al. · similarity 0.88 SIMA 2: A Generalist Embodied Agent for Virtual Worlds We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds. Built upon a Gemini foundation model, SIMA 2… Bolton et al. · similarity 0.87

Related notes

References

  1. See and Think: Embodied Agent in Virtual Environment (arXiv:2311.15209)
  2. Voyager: An Open-Ended Embodied Agent with Large Language Models (arXiv:2305.16291)
  3. Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288)