An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

Pixel-space diffusion has a genuine appeal: no VAE bottleneck, no reconstruction ceiling, and at inference time you skip the decoder pass entirely. The problem is that nobody had rigorously shown how to make it work at scale for text-to-image generation. This paper does exactly that, and the central finding is uncomfortable for pixel-space enthusiasts: training directly in pixel space, even on more than 20 billion image-text pairs, converges substantially more slowly than an equivalent latent-space run. The VAE isn't just a compressor — it's a learned representation that hands the diffusion model a much simpler distribution to work with. Pixel space forces the model to simultaneously discover global structure, local statistics, and fine-grained detail from raw RGB signals. That's a harder optimization problem, full stop.

The practical answer the paper arrives at is a latent-to-pixel strategy: pre-train in latent space, then adapt to pixel space during post-training. But the adaptation itself requires careful engineering across five dimensions, and the paper works through each one systematically. Weight initialization from the latent checkpoint matters enormously — starting from scratch in pixel space is dramatically slower. Training data composition matters too: self-generated samples from the same source model converge fastest because they stay close to the model's learned conditional distribution, but training on them alone inherits the source model's artifacts, including malformed typography. Mixing self-generated samples with high-quality real images at a one-to-one ratio fixes this.

The choice of prediction target is also non-trivial. Prior work suggested x-prediction was essential for stable pixel-space training, but here latent initialization stabilizes v-prediction as well — and v-prediction consistently scores better on benchmarks. For the decoder head, a lightweight convolutional U-Net (DiP) beats both a linear projection and a much heavier Transformer-based head on the quality-efficiency curve; the Transformer head carries a decoder alone that is more than a third the size of the backbone, which is simply impractical. The noise schedule requires empirical calibration: a theoretically motivated resolution-based value turns out suboptimal, and a different empirically chosen scale produces noticeably better color rendition.

Efficiency gains compound from two additional steps. Progressive patch-size adaptation — first adapting from latent to pixel at a fine token grid, then coarsening to a larger patch size — decouples the two transitions and avoids the artifacts that appear when both happen simultaneously. Step distillation then removes the fixed VAE decoding cost that limits latent-space distillation in the ultra-few-step regime. The combined result is an end-to-end speedup in the range of roughly three to nearly five times over the latent-space baseline, validated on two separate model families (Z-Image and FLUX2-klein), which is the paper's strongest evidence that the recipe generalizes.

What's useful here isn't any single trick but the disciplined ablation structure. Each design choice is isolated and measured, and the interactions between them are made explicit. For anyone building production image generation systems where inference cost is real, this is a credible engineering map.