OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

The core problem OmniAssistBench addresses is deceptively simple to state: when an AI assistant tells a user to do something, the user does it, and the video changes. Every existing benchmark ignores this. Static question-answer pairs over fixed videos cannot capture the causal loop where a model's response shapes what happens next. The benchmark's solution is elegant if labor-intensive — fix the interaction path in advance by extracting procedural priors from the source video, then reverse-engineer that video into multi-turn clips. The model is never free to invent an alternative valid route; it must follow the one the video already shows. This sidesteps the combinatorial explosion of diverging paths without pretending the problem doesn't exist.

The construction cost is real: over a thousand expert person-hours, entirely manual annotation, and three purpose-filmed real-world cases averaging fifteen interaction turns each. The decision to reject automated annotation pipelines is principled — the tasks deliberately target non-salient details that vision-language models routinely omit from captions, and the user goals in the advanced tier require human judgment to verify logical coherence with the video content.

The results are sobering. Gemini-3-Pro leads at 66.4 out of 100; the best open-source model, Qwen3-Omni-Instruct at 30 billion parameters, reaches 51.2. Four failure modes recur across models. Gesture-based instructions trip up nearly everyone — the benchmark treats hand gestures as first-class input modalities, and current training data simply hasn't prepared models for them. Context windows collapse under sustained interaction: a model with a 32k window can hold roughly 380 seconds of video at one frame per second, while the Qwen family's denser frame encoding shrinks effective memory to around 80 seconds. Neither is enough for a twenty-minute meeting simulation. Models also struggle to delay responses — when told to wait for a specific event before answering, they caption whatever is currently on screen instead. And cross-turn coherence degrades badly: later visual inputs cause models to forget goals stated earlier.

The modality ablation surfaces something counterintuitive. On the proactive response task, stripping audio actually improves scores. With audio present, models mistake background speech in the video for new user commands and abandon the original goal. Silence, paradoxically, helps them stay on task.

The teacher-forcing experiment — replacing prior model outputs with ground-truth answers to eliminate error propagation — shows only modest gains. Models adapt to the ground-truth format but still fail to connect those correct answers to the ongoing video. The bottleneck is not error accumulation; it is genuine comprehension failure.

For anyone building agents that operate in continuous, embodied, or procedural environments, the benchmark's taxonomy of failure is the most actionable part: gesture blindness, memory collapse, premature response, and goal drift are four distinct engineering targets, not one vague capability gap.