Hex color codes work as text prompts once you finetune the denoiser, not the encoder

Most image generation models treat color as a suggestion. Ask for a specific shade and you get something plausible, not precise. Paint-Anything attacks this directly: it teaches a diffusion model to honor exact 24-bit hex values, the same specification format a brand designer or product team would actually use.

The core insight is that large language models already understand hex strings. Even a compact 4B model can map raw codes like #F0FFF1 to coherent color semantics, which means hex can serve as a text-native interface rather than requiring a separate color encoder or inference-time guidance mechanism. Paint-Anything exploits this by wrapping hex values in explicit <color> tags inside ordinary prompts and finetuning the denoising transformer while keeping the text encoder and VAE frozen.

The training data problem is harder than the architecture problem. Real photographs are lit by the world: shadows, highlights, and material properties mean that a single-colored object spans a wide range of pixel values. The paper's solution is two-pronged. First, a 500K-sample dataset is built by running VLM grounding and SAM3 segmentation on real images, then extracting dominant colors using MeanShift clustering in CIELAB space rather than fixed-k RGB k-means. MeanShift adapts the cluster count to each object's actual color distribution, reducing the redundant splits that plague fixed-k approaches. Second, pure-color anchors—solid-color images paired with their exact hex codes—provide a clean low-level signal. Crucially, these anchors are gated to high-noise timesteps only, because color stabilizes early in the generation trajectory. Leaving low-noise training to natural images preserves realistic appearance while the anchors handle numeric grounding. The ablations are specific: gated anchors improve ACBench-T2I by 4.75 points and ACBench-Edit by 5.18 points over ungated anchors.

The benchmark results are worth examining carefully. On FLUX.2-4B, finetuning improves ACBench-T2I by 85.3% and ACBench-Edit by 28.3% relative to the base model. More striking is the cross-scale comparison: the finetuned 8B model exceeds the 56B off-the-shelf FLUX.2 variant by 16.88 points on generation and 6.70 points on editing. The paper also documents a word-to-hex gap in the base model—replacing color names with hex strings in CompColor prompts drops the base model's average score from 0.72 to 0.38. After finetuning, the hex-prompt average more than doubles and exceeds the base model's named-color average, while the named-color average itself improves from 0.72 to 0.79.

The reliability appendices are unusually thorough. Rankings hold across five alternative color estimators, a sweep of SAM3 detection thresholds, and a localization-failure audit. The human preference study is modest—15 participants, 80 prompts—but directionally consistent: Paint-Anything is preferred on color fidelity in 55% of trials versus 13% for the base model.

The stated limitation is narrow: the training data lacks palette-level supervision, so multi-color coordination across an image is out of scope. The internal source collection is not publicly available, which constrains reproducibility of the data pipeline even if the model weights are released. These are real constraints, not minor caveats.