Bidirectional cross-attention in joint video models is secretly one-directional

Joint multimodal diffusion transformers are architecturally bidirectional but functionally lopsided. When a model generates video alongside 3D human motion or audio, both modalities can attend to each other through cross-attention — but in practice, the video-to-companion pathway learns strong, spatially coherent correspondences while the reciprocal companion-to-video pathway stays weak. The video stream dominates, and the companion modality fails to meaningfully constrain it.

This paper names that asymmetry the reciprocal correspondence gap and proposes a fix called RecCAR — Reciprocal Cross-modal Attention Regularization. The core idea is elegant: both attention directions describe interactions between the same token pairs, just viewed from opposite sides. Since the video-to-modality direction already encodes useful correspondence structure, it can serve as a supervision signal for the weaker direction. RecCAR re-normalizes the modality-to-video attention logits over video tokens (making both directions answer the same question: which video tokens correspond to this companion token?), then applies a stop-gradient KL objective that pulls the weaker distribution toward the stronger one. No external labels required. No architectural changes. Just LoRA fine-tuning on the existing cross-attention weights.

The ablations are the most convincing part. On video–motion generation, fine-tuning EchoMotion on the same curated data without the RecCAR loss actually degrades human anatomy scores and increases pose error. Adding RecCAR flips that: Human Anatomy improves from 0.69 to 0.75, and pose alignment tightens substantially — without reducing dynamic degree or motion smoothness. The gains are specifically from the alignment objective, not from more data or more compute.

The audio–video results follow the same pattern. Applied to LTX-2, RecCAR reduces absolute audio–video desynchronization from 0.804 to 0.752 on T2AV-Compass. Standard fine-tuning on the same VGGSound clips moves the needle almost nowhere. The trotting-horse example in the paper is illustrative: the baseline produces hoofbeat transients that lead or lag visible hoof contacts; RecCAR tightens that temporal coupling while preserving audio realism scores.

The mechanistic validation is careful. The paper directly measures the correspondence gap before and after training, per test prompt. The vast majority of prompts fall below the diagonal — RecCAR reduces the gap consistently across the test set, not just on average. Attention mass also shifts: the reciprocal modality-to-video direction draws more weight from video tokens, with the two population clusters cleanly separated and non-overlapping.

The training setup is modest: roughly 4,300 video–motion pairs filtered by joint MPJPE and NMPJPE thresholds, 4,300 VGGSound clips for audio, LoRA rank 128, 10 epochs, 4 H100 GPUs for about 48 GPU-hours per track. The filtering pipeline itself — using Gemini anatomy ratings to calibrate a pose-error threshold on 25 videos — is a practical detail worth noting for anyone building similar training sets.

The broader point lands clearly: coupling two generative streams through bidirectional attention is not sufficient if one modality learns to dominate. The fix does not require redesigning the architecture.