Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs

Hybrid-thinking multimodal models let a single deployment switch between a deliberative reasoning mode and a faster direct-answer mode. The implicit promise is that both modes deliver responses of equivalent quality to the user. This paper tests that promise and finds it consistently broken.

The mechanism of failure is straightforward: non-thinking inference, stripped of the extended reasoning budget, produces responses that leak deliberation-style content, repeat themselves, contradict their own claims, or stage analysis without supporting evidence. The authors call these four patterns CoT leakage, response repetition, logical contradiction, and performative reasoning. They build PatternEval — 2,415 multimodal prompts across visual perception, OCR and structured-image understanding, and knowledge reasoning — specifically to stress-test these failures under matched thinking and non-thinking conditions.

The results across 25 model configurations are striking in their uniformity. Every single evaluated pair shows a higher failure rate under non-thinking inference. The gap exceeds 20 percentage points for 17 of the 25 pairs. The most extreme case, Kimi-K2.6, shows a non-thinking trigger rate of 48.64 percentage points above its thinking-mode counterpart. CoT leakage is the dominant failure: averaged across all 50 model-mode rows, it triggers at 12.75%, compared with 8.49% for repetition, 4.72% for performative reasoning, and 2.74% for contradiction.

Scale does not reliably close the gap. Within the Qwen3.5 series, thinking-mode trigger falls steadily from 22.19% at 4B to 4.23% at 397B-A17B, but non-thinking trigger drops from 46.42% at 4B to around 29% at 27B and then plateaus, leaving the cross-mode gap roughly stable between 21 and 27 percentage points across the larger models. Stronger task accuracy and a lower failure rate are not the same thing.

Response length compounds the problem. Longer responses correlate with higher failure rates under both modes, with Pearson correlations of roughly 0.74 under non-thinking and 0.83 under thinking inference. Within the longest response sextile, incorrect non-thinking responses trigger failures at the highest observed rates — but even long, correct non-thinking responses show elevated failure rates, which means correctness alone does not guarantee a well-formed delivered response.

The paper's training contribution, PatternRL, adds a reward model (PatternRM, initialized from Qwen3.5-27B and trained on 57,578 supervised examples) that penalizes the four failure types during reinforcement learning. On Qwen3-VL-4B and Qwen3-VL-8B, PatternRL reduces non-thinking trigger by 13.08 and 14.35 percentage points relative to a correctness-only baseline, while aggregate accuracy shifts by less than one percentage point. The trade-off is capacity-dependent: the 4B model takes a more visible accuracy hit, particularly on math reasoning, because the pattern penalties restrict exploratory, verbose trajectories that smaller models rely on to reach correct answers.

The residual failures after PatternRL are honest about what RL-stage intervention can and cannot fix. Behaviors baked in during earlier training stages persist. The authors suggest that durable improvement requires response-pattern constraints at the supervised fine-tuning or mid-training level, not just as a late-stage reward signal. That is a practical research direction, not a caveat — and it is the most useful thing this paper leaves open.