A single omni-modal encoder now outperforms modality-specialist assemblies on retrieval

Most multimodal embedding systems are assemblies: a text backbone with a vision tower bolted on, and then an audio encoder aligned to whatever geometry that pair established. Ovis-Embedding takes a different route. It starts from Qwen2.5-Omni, a model that already processes text, images, video, and audio through a single shared Transformer called the Thinker, then strips the speech-generation head and reads the final hidden state at the last non-padding token as the embedding. No modality-specific projection heads are added. The geometry the backbone learned during pretraining is the geometry the embedder inherits.

The training recipe has four stages. Stage one uses LoRA to stabilize early contrastive pretraining—the paper makes a pointed observation here: full-parameter updates at initialization, before the embedding space has any coherent structure, let unstable gradients erode the pretrained semantics before representations stabilize. Stage two unfreezes everything but switches to homogeneous-source sampling, where every micro-batch is drawn from a single dataset. The motivation is concrete: mixed batches let the model exploit modality or format cues as shortcuts rather than learning fine-grained semantic distinctions. Stage three is Embedding Distillation, which filters training examples to those the teacher solves correctly, upsamples the student's failures, and uses forward KL divergence to transfer the teacher's full ranking distribution rather than a one-hot positive label. Stage four fits a lightweight residual adapter on top of a shared PCA basis to produce nested embedding widths from the same frozen encoder without retraining.

The numbers are competitive. Ovis-Embedding-Omni-3B scores 58.46 on MMEB-v3, beating the next-best model by over five points, and leads all six modality groups including audio, where the margin is more than seven points. On MAEB it reaches a Mean(Task) of 57.29 and on MVEB 61.77, outperforming a 7B competitor despite the smaller parameter count. The VL-9B variant scores 81.13 on MMEB-v2, and the 2B variant 77.46. On RTEB, the omni model slightly edges a dedicated text-embedding model at 4B parameters, which is a useful signal that broader modality coverage does not degrade text retrieval.

The elastic embedding results are honest about limits. The paper notes that a sixteen-fold dimension reduction still retains most performance, but the losses are uneven across modalities. VisDoc and Agent tasks suffer the most at short widths because they require fine-grained discrimination—localizing a passage within a page, or distinguishing interface states that differ in small elements—and those decisions depend on lower-variance directions that short prefixes discard. Audio and video are more tolerant because their queries are largely settled by coarse category-level evidence that survives in the leading directions.

The one acknowledged weakness is multi-condition text retrieval, where the model trails a competitor by nearly eight points. That gap is narrow enough to be a roadmap item rather than a structural flaw, but it is real.