Per-vGrid token grouping and reward-routed RL make live-commerce multimodal AI viable

E-commerce live streaming is a genuinely hostile environment for multimodal models. Product facts scatter across host speech, overlaid text, video frames, and product images simultaneously. A host mentions a brand name once, quickly, while a price graphic flashes on screen. General omni-modal models were not built for this, and TLive-Omni is an explicit attempt to fix that gap.

The architectural centerpiece is Per-vGrid, a token organization scheme that groups each video grid with the audio covering the same time interval, bounded by explicit delimiter tokens and a textual timestamp derived from actual sampled frame indices rather than the requested sampling rate. The distinction matters: a 119-frame video at 30 FPS sampled at roughly 2 FPS yields realized frame indices that shift grid timestamps and audio-token spans by one or two tokens compared with naive rate assumptions. That precision is what makes temporal alignment tractable across long streams.

The model grafts Qwen3-Omni's audio transformer encoder onto a Qwen3.5 vision-language backbone through a lightweight aligner, supporting up to 256K tokens of multimodal context. The audio encoder compresses speech to roughly 13 tokens per second, which keeps long recordings feasible within that budget. Training proceeds in three supervised stages—audio-language alignment on 5M ASR samples, broader audio understanding on 26M audio samples, then joint multimodal fine-tuning on 14M samples spanning all modalities—before a reinforcement fine-tuning stage called Faithful-RFT.

Faithful-RFT is the more interesting methodological choice. Rather than rewarding visible chain-of-thought reasoning, it suppresses explicit think traces and scores final answers directly using task-conditioned reward routing. Different tasks—visual grounding, OCR, open-ended video QA, ASR—each get their own reward functions; inapplicable ones return a sentinel and are excluded before aggregation. A dynamic resampling strategy discards rollout groups with near-zero reward variance and regenerates them, ensuring GRPO receives informative relative-advantage signals rather than flat gradients.

On the live-commerce evaluation suite, the 9B variant achieves the lowest character error rate on ASR, the highest product visual grounding AP on product images, the highest temporal grounding mIoU, and the lowest dense caption hallucination rate among evaluated open-source models. The 4B variant leads open-source results on all three TimeLens-Bench temporal grounding splits. On general benchmarks, the 9B model leads open-source results on MLVU, Video-MME, LongVideoBench, MMVU, and four omni-modal benchmarks.

The paper is candid about scope: TLive-Omni handles understanding only, not generation or full-duplex real-time interaction. Calibration under incomplete or ambiguous multimodal inputs—common in actual live streams—is flagged as unsolved. The evaluation suite is in-house, which limits external reproducibility of the live-commerce numbers specifically.