SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

The best video generation model tested here achieves an Outcome Achievement score of 37.8%. That number is the central finding of SemComp-Bench, and it is damning.

The benchmark targets a specific gap: existing evaluations measure visual fidelity, temporal coherence, and prompt adherence, but none jointly test whether a video actually reaches an instructed outcome while staying semantically grounded in a reference image. SemComp-Bench fills that gap with two scores. The OA Score is conjunctive — a video must pass all four criteria (outcome realization, semantic grounding, grounded entity consistency, global visual continuity) simultaneously, which is why even strong models score low. The GR Score averages five reliability criteria independently of task completion.

The dataset, SemComp-Data, contains 1,273 instances drawn from Koala-36M, organized into six domains across 21 task categories. Each instance pairs a reference frame with an outcome-centric clip extracted from the same source video, which is the key design choice: because the reference and outcome come from one real recording, every task is demonstrably achievable and visually verifiable. A four-stage curation process handles filtering, frame-pair localization, clip extraction, and instruction generation — producing both a brief and a detailed instruction per instance from a template constrained to 30 words.

The experimental results reveal a clean dissociation between reliability and task completion. Seedance 2.0 leads on GR at 91.8% but performs substantially worse on OA. HunyuanVideo-1.5-720P-I2V leads on OA at 37.8% despite a lower GR ranking. Only Wan2.2-I2V-A14B ranks second on both, suggesting that balanced performance across the two dimensions is rare. Within-scene spatiotemporal coherence is the single worst reliability criterion across all models, with pass rates ranging from 0.328 to 0.739 — models can generate convincing individual frames but struggle to maintain stable visual evolution within a scene.

The instruction specificity findings are counterintuitive. Detailed instructions improve outcome realization and semantic grounding but hurt grounded entity consistency and global visual continuity. Brief instructions produce more coherent videos that less often achieve the intended outcome. More specification creates more failure modes.

Image conditioning matters enormously. I2V variants consistently outperform T2V counterparts, with gains concentrated in semantic grounding and entity consistency rather than outcome realization — which is roughly comparable across conditioning modes. The reference image is load-bearing for identity and structural preservation, not for knowing what the finished state should look like.

The paper is honest about what remains unvalidated: whether SemComp-Data can improve model performance through task-specific training is explicitly left as future work. The evaluation itself relies on a single VLM judge (Doubao-Seed-1.8) across all scoring calls, which is a real dependency even with three independent runs per video.