Keeping just 5% of KV positions in full precision makes 4-bit attention viable

Attention maps in modern LLMs are not uniform. A small fraction of key positions—often around 5%—consistently absorbs a disproportionate share of attention mass across many query tokens, forming persistent vertical stripes in the attention heatmap. HyQuant exploits this directly: keep those positions in full precision, quantize everything else to 4-bit, and fuse the two paths into a single FlashAttention-style kernel.

The motivation is quantitative, not just visual. On Llama-3.1-8B and Qwen3-8B, the top 5% of key positions plus a short local window already cover the substantial majority of total attention mass. More tellingly, an MSE analysis on Qwen3-8B shows that uniform 4-bit quantization produces noticeably larger errors than 8-bit, but retaining only the top 1–5% of high-score positions in full precision while keeping the rest at 4-bit brings the error back down to near 8-bit levels—across sequence lengths from 1K to 32K.

The design splits cleanly across the two inference stages. During Prefill, which is compute-bound, HyQuant runs most of the attention computation in low precision while keeping vertical-line tokens and a local sliding window in FP16/BF16, fused into one kernel. During Decode, which is memory-bandwidth-bound, the KV cache is stored in hybrid format—4-bit for the bulk, full precision for the critical set—and dequantization happens on the fly inside the attention kernel rather than as a separate materialization step. That fused design is what lets HyQuant operate at batch size 32 on a single H100-80GB where competing methods like KIVI and KVTuner run out of memory.

Speedup numbers are honest about where the gains actually live. Decode kernel speedup ranges from 1.32× to 3.58× over FlashAttention-2 depending on prefix length, with larger gains at longer contexts where memory bandwidth is the bottleneck. End-to-end decode speedup is more modest: 1.04× to 1.17×. Prefill latency is comparable to SageAttention; the paper claims accuracy improvement there, not speed. The overhead for identifying vertical-line tokens—accumulating column-wise attention scores every 64 tokens—runs to only 3–5% of total runtime.

The key distinction from MInference, which also identifies vertical-line patterns, is that MInference uses them as a sparsity mask and discards the rest. HyQuant keeps non-vertical tokens in low-bit form, so their information is not lost—just compressed. The ablation confirms this matters: dropping the quantized long tail hurts benchmark scores on LongBench tasks.

Limitations are stated plainly: benefits are most visible at long context; short-context gains are smaller. Evaluations are on H100 only, and no model larger than Qwen3-32B was tested. Whether the approach holds in agent or coding workloads is explicitly left open.