GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Video games have long been a proxy for AI capability, but the benchmarks built around them have been narrow, fragile, or both. GameHorizon addresses this directly by building a three-part suite: an automated annotation tool, a large-scale dataset, and a benchmark with both offline and online evaluation tracks.

The annotation approach is the most technically interesting piece. Rather than relying on visual scene detection—which over-segments under rapid camera movement—the system uses synchronized keyboard-and-mouse traces to find action boundaries. A VLM then disambiguates cases where the same input carries different meanings depending on context (a left click that might be an attack or an item selection). Instructions are built bottom-up across three temporal scales: short-horizon operations spanning roughly seconds, medium-horizon goals spanning minutes, and long-horizon strategies spanning longer stretches. Each frame ends up aligned to all three levels simultaneously.

The resulting dataset covers multiple AAA game titles across genres including open-world, competitive shooter, and creature-collecting adventure. It contains millions of keyboard-mouse action events and a large number of distinct multi-horizon instructions, with one new short-horizon instruction generated roughly every few seconds of footage—substantially denser than prior work like Open-P2P, which provided one instruction every few minutes.

The benchmark's offline track uses multiple-choice questions organized into three primary tasks: single-horizon action prediction, multi-horizon goal decomposition, and cross-horizon consistency. The difficulty ordering is coherent: action prediction is hardest, consistency is easiest, and decomposition sits between them. Across more than a million model invocations, proprietary general-purpose VLMs dominate the top tier, with GPT-6-Astra leading at the highest overall accuracy. Dedicated game agents, by contrast, score near random on unseen AAA titles—JARVIS-VLA drops sharply when moved off its Minecraft training domain.

Several findings are worth noting for anyone building agents. First, thinking-mode inference helps capable models but actively hurts smaller ones, apparently because lightweight models generate hallucinated reasoning chains that mislead action prediction. Second, future-action planning consistently lags current-action perception by a meaningful margin across all input settings—models are better at recognizing what just happened than deciding what to do next. Third, adding medium- and long-horizon instructions to the input improves future planning by more than adding short-horizon instructions alone does, which validates the multi-level annotation design.

The online track, implemented in Minecraft for accessibility reasons, confirms that offline scores predict online gameplay rankings: models from each offline tier occupy the corresponding online tier positions. Long-horizon task completion remains difficult even for top models, with the best achieving a modest success rate and most models at or below that.