Dynamic Multi-Byte Prediction With Hierarchical Language Models

Byte-level language models have a well-understood appeal: no fixed vocabulary, no tokenizer bias, no fragmentation of rare words. The persistent problem is that generating one byte at a time is slow, and the longer sequences that byte-level models produce make that slowness worse. LCA-MBP attacks this directly by reusing the segment structure that hierarchical byte-level models already learn, turning it into a unit of parallel generation rather than just a compression device.

The core mechanism is a custom attention mask called Latent Causal Attention. Within a predicted segment, each byte position can attend to itself, to earlier positions in the same segment, and to all bytes in the immediately preceding segment—but not to later positions in the same segment and not to segments further back. This means all bytes in a segment can be predicted in a single forward pass without violating autoregressive causality at the segment level. Crucially, this is accomplished with a single decoder head rather than one head per future token, which is how prior multi-token prediction methods scale their parameter count.

The paper trains a 373M-parameter model on 50 billion bytes from FineWeb-edu and evaluates across instruction following, question answering, summarization, and machine translation. LCA lands on the Pareto front of performance versus throughput on three of the four tasks. The fourth, Spanish-English translation, sees a small quality drop relative to the strongest non-MBP baselines, but those baselines run at substantially lower throughput.

The comparison against MLP-based multi-byte prediction is instructive. MLP heads achieve higher raw acceptance rates—they are confident—but that confidence is misplaced: downstream quality is worse than LCA at comparable throughput. The transformer layers in the LCA head condition each predicted byte on the previous segment's context and on its position within the current segment, producing more coherent predictions than independent MLP heads operating on the same shared hidden state.

The speculative decoding mode is the most practically interesting result. When LCA's multi-byte predictions are verified by an external model rather than accepted by threshold, quality is guaranteed to match the verifier by construction. Under this regime, increasing the number of candidate bytes becomes a free throughput knob—speed scales monotonically while output quality is preserved. Using LCA as a drafter for an external FxT verifier of the same size yields a 1.4 to 1.7 times speedup across three tasks, which is notable because the speedup comes not from a smaller draft model but from predicting an entire latent token's worth of bytes in parallel.

The limitations are stated plainly: experiments run at a single model scale, the evaluation is English-centric, and the optimal speculative horizon is task-dependent under threshold acceptance (though not under speculative decoding verification). Whether the gains persist at larger scales is explicitly left open.