NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

Most video benchmarks treat "long" as a duration threshold and call it done. NARU makes a more useful distinction: a four-hour surveillance loop is long, but it does not require a model to track who changed their mind, why a host's coffee offer signals the end of a visit, or how a documentary's central argument shifts across its runtime. The benchmark calls this difference context-rich long-form video, and it is the right cut.

The dataset is 155 Japanese YouTube videos totalling roughly 146.8 hours, yielding 1,481 multiple-choice questions across four narrative and five cultural dimensions. The cultural taxonomy is where the work gets specific. Categories include aizuchi — the short backchannels that can signal agreement, surprise, or turn-management depending on prosody and timing — and kuuki wo yomu, the inference of unspoken group atmosphere from distributed behavioral cues. These are not exotic edge cases; they are load-bearing features of Japanese conversation that a model cannot fake with surface pattern-matching.

Construction at this scale required a hierarchical pipeline: five-minute chunks processed sequentially with a rolling narrative history, then a second pass for semantic segmentation, then taxonomy-aligned annotation modules, then a Solver-Critic loop that iteratively rewrites questions a blind agent can answer without watching the video. Two separate cohorts of native-speaker annotators — 40 in the first pass, 28 in the second — verified grounding and cultural fidelity. The result is a benchmark that is genuinely hard to game.

The evaluation numbers confirm the difficulty. Gemini-3-Flash leads all models at 76.2% on multiple-choice, but drops to 57.4% on subtext interpretation (C.3), the category requiring inference of a speaker's private intent behind a socially restrained utterance. Open-source models cluster between roughly 30% and 40% overall, with several falling below the 25% random-chance baseline on entity tracking and cultural tasks. Feeding more frames helps narrative performance substantially — Gemini-3-Pro climbs about seven percentage points moving from 8 to 128 frames — but cultural accuracy barely moves, which suggests the bottleneck there is pre-training knowledge coverage, not temporal resolution.

The open-ended evaluation adds a further wrinkle. Sequential flow (N.2) is the easiest narrative category under multiple-choice but degrades to the weakest for most models when answer choices are removed. The choices were acting as scaffolding for temporal reconstruction, not evidence of genuine understanding. That reversal alone is worth the price of the benchmark.