Stripping vision from a Mario AI reveals a sharper way to feed state to language models

The interesting design choice here is what gets removed from the model's view. Jev never sees a screenshot. Instead, a RAM parser extracts structured telemetry from the emulator and hands the model a compact JSON object organized around meaning: where Mario is, what his jump trajectory looks like, where enemies are projected to be, how much time elapsed between the last observation and the action that followed. The raw tile grid exists in debug logs and the dashboard, but it is explicitly excluded from model input. That exclusion is a thesis: vision is not the right interface for this problem.

The action vocabulary is deliberately narrow — seven macros covering the essential movement grammar of World 1-1. No fine-grained button combos, no frame-perfect inputs. The model picks one macro every eight emulator frames, which is a coarse cadence by any speedrunning standard but reasonable for a language model operating over structured state.

What makes the architecture worth studying is the three-judgment structure. Each decision cycle runs a Choice for the controller macro, a Noul to estimate whether a forward jump is currently useful, and a Score to quantify immediate danger. These are independent evaluations over the same state object, not a single prompt asking the model to do everything at once. The timing arithmetic — combining measured response latency, enemy velocity, action duration, and jump clearance time into a single typed boolean called jump_must_start_this_decision — happens in Python before the model ever sees the state. Jev interprets facts; it does not compute them.

That division of labor is the real argument the repo is making. The harness does the geometry. The model does the judgment. There is no scripted override that kicks in when the model hesitates — the README is explicit that Jev owns the controller choice unconditionally. Whether that produces competent play is an empirical question the repo leaves open; it logs every decision to a timestamped JSONL file with latency, action probabilities, confidence, and game outcome, so the data is there for anyone who wants to measure it.

The dashboard is more thoughtful than most hobby projects bother with: live probability distribution, confidence, latency, jump probability, danger score, and parsed game state all visible in a single recordable window. The state-demo command lets you inspect exactly what JSON and text would be sent to the API without launching the game or spending any API credits, which is a small but genuinely useful affordance for debugging prompt design.

The broader question this raises for agent builders is whether object-centric structured state is systematically better than raw observation for model-driven control — and whether the answer changes depending on how much domain geometry you are willing to pre-compute in the harness.