Scoring logits beats generating JSON for multi-question visual inference

The core idea here is simple and worth understanding: instead of asking a vision-language model to generate free text and then parsing the result, you encode the image once, cache that shared context, and then score a fixed set of candidate tokens directly from the model's logits. One image, many questions, no autoregressive generation per question. The approach sidesteps structured-output fragility entirely - the README notes that generated JSON failed schema validation at 4, 16, and 64 decisions in benchmarks, which is exactly the failure mode this pattern avoids.

The performance numbers are concrete. On an M4 Mac with 16GB RAM, processing 64 decisions via independent candidate scoring took a median of 37.30 seconds. The shared-prefix batched approach took 2.40 seconds for the same workload. That's not a marginal improvement; it's a different scaling curve. The gains come from reusing the KV cache across questions rather than re-encoding the image and shared context for each one.

The model in use is Qwen3.5-0.8B at 4-bit quantization, weighing in at roughly 596 MiB of downloaded weights. The README is honest about what this size can and cannot do. The Breakout demo - where the model controls a paddle by classifying which of five screen regions contains the ball - required deliberate simplification: larger ball, wider paddle, slower mode, and a region-classification framing rather than direct left/right commands. In a recorded test run, 80 decisions yielded 9 bricks cleared and 6 returns with 2 lives remaining before the test was stopped. The README explicitly calls this a simplified demo, not evidence of general game-playing ability, and notes that 4-bit quantization effects were not isolated.

The inference path is laid out in reading order across four files: preprocessing, adapters, scoring, and engine. Cache sharing is within a single request and copies state rather than sharing zero-copy - a limitation stated plainly. Candidate probabilities are relative to the supplied options, not absolute correctness estimates. The repo supports 1 to 64 questions with 2 to 26 options each.

This is explicitly a learning project for Apple Silicon, not a production serving system, and it says so. It credits OpenJev and a Hugging Face RLCD model for the candidate-scoring and shared-context ideas. The value here is pedagogical and architectural: a runnable, benchmarked demonstration that logit-scoring over a shared multimodal prefix is both faster and more reliable than generation-then-parse, at least at this scale and on this hardware. Anyone building multi-question visual classifiers on local hardware has a working reference implementation with honest performance data attached.