WeMM-Embedding leads multimodal retrieval benchmarks while skipping audio entirely

WeMM-Embedding is a family of three multimodal embedding models — 2B, 4B, and 9B parameters — that produce a single vector space covering text, images, videos, visual documents, and interleaved mixed inputs. The embedding is extracted from the last hidden state at a dedicated <embedding> token, then L2-normalized. Audio is explicitly not supported.

The design choice worth pausing on is Matryoshka Representation Learning across all three model sizes. Every model supports a range of truncated dimensions, from 64 up to 4096 depending on the variant. The README reports that the 2B model at just 256 dimensions retains 98.7% of its full-dimensional performance on MMEB-v2 for image and video tasks. That is a meaningful compression story for anyone building retrieval systems where vector storage or ANN index size is a real constraint.

On MMEB-v2, which covers 78 datasets, the 2B variant scores 77.9 average, ahead of Qwen3-VL-Embedding at 73.2 and DME-Small at 74.8 — the latter being a closed-source submission with no public weights. The 9B model reaches 80.6, above the comparable 9B DME-Medium at 78.4 and Qwen3-VL-Embedding at 77.8. Those are not marginal gaps.

MMEB-v3 is the more revealing benchmark. It expands to 190 tasks including agent tasks, multi-modal cross-modal retrieval (MCMR), and audio. WeMM-Embedding scores zero on audio across all sizes — the README is direct about this — yet still leads the V3-All ranking at 59.5 for the 9B model, because its text, agent, and MCMR scores are strong enough to compensate. The 9B model scores 51.0 on agent tasks and 49.3 on MCMR, both the highest in the table. For anyone building retrieval pipelines that need to handle agent trajectories or cross-modal matching, those numbers are the ones to watch.

Serving is covered for both vLLM and SGLang, with specific tested versions called out. The evaluation code ships as a modified fork of the VLM2Vec pipeline with multi-node torchrun support and 64-frame video sampling. The README is honest about what changed from upstream, which makes it easier to trust the reported numbers.

The practical question for builders is whether a single embedding space that handles video and agent tasks at this quality level changes what retrieval-augmented systems can index. The answer looks like yes, with the caveat that audio retrieval requires a different model entirely.