Sorting value tokens beats rotating them for low-bit video diffusion attention

Low-bit attention for video diffusion has a solved problem and two unsolved ones. The solved problem is the score product: smoothing queries and keys, adding Hadamard rotations, and per-block scaling have collectively driven probability error low enough that it no longer dominates. The two unsolved problems are value quantization and softmax speed, and VC-Attention addresses both without retraining.

Value outliers in video diffusion transformers sit in tokens, not channels. That distinction matters because every prior fix—channel mean subtraction, Hadamard rotation—operates on channels. A rotation preserves token norms, so an outlier token survives it intact and still sets the quantization scale for its entire hardware block. On Wan2.2, after all existing query-key smoothing is applied, the value term accounts for 82% of the remaining output error.

V-Smooth's response is to reorder tokens before quantization. An online k-means groups value tokens by similarity, then sorts them so that each hardware block holds tokens that share a common component. Subtracting the block mean before quantizing leaves only the residual for the low-bit quantizer. The mean itself costs almost nothing to restore: the online softmax recurrence already accumulates the row sum needed to recover it, so no extra pass or buffer is required. Averaged over 100 Wan2.2 heads, sorting raises the share of block energy removed by mean subtraction from 8% in sequence order to 36%. The PSNR improvement over SageAttention2 is 2.3 dB on Wan2.2 and 2.8 dB on HunyuanVideo-1.5 at 8 bits, and 2.9–3.6 dB over SageAttention3 at 4 bits.

The second problem is that on B200, Tensor Core throughput has doubled relative to Hopper while the multi-function unit still issues 16 exponentials per SM per clock. Once both matrix products run in FP8, the FP32 exponential and the FP32-to-E4M3 cast become the longest pipeline stage. ExpCast-FP8 eliminates both by observing that an E4M3 byte is already a logarithmic encoding of its value, so the probability code is an affine function of the log-domain score the online softmax already holds. One fused multiply-add produces the byte directly. The row-level total variation error is bounded at 3.64% plus any underflow tail, and empirically averages 1.6% across 204.8K attention rows.

Grouping runs only on the first quarter of denoising steps and reuses each permutation for four adjacent steps, since attention layouts change slowly between steps. Averaged over a full generation, grouping costs 3–4% of attention time. The full fused preprocessing chain—rotary embedding, smoothing means, gather, Hadamard, quantization—is hand-written in CuTe/CUDA, yielding an 8.74× reduction in preprocessing time over the unfused chain on B200.

The kernel speedups over BF16 FlashAttention-4 are 1.59× on B200, 1.46× on H200, 2.27× on the RTX PRO 6000, and 3.58× on the RTX 5090. End-to-end clip generation is 1.19× faster on B200 and 1.70× faster on the RTX 5090. The workstation gains are larger because FP4 Tensor Cores on those cards offer an 8× peak over BF16, and V-Smooth's fidelity advantage over SageAttention3 is where the two methods actually differ at 4 bits.