EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Free-form chain-of-thought re-ranking has a structural problem: it omits implicit constraints, hallucinates visual properties, and covers semantic dimensions inconsistently across queries. EviRank addresses this by treating multimodal image re-ranking as constraint satisfaction rather than similarity matching.

The core idea is an Evidence Frame - a typed package of short, checkable sentences organized across six semantic slots (entities, attributes, actions, relations, scene, key details), each labeled required, forbidden, or ignorable. Forbidden constraints are written as affirmative statements describing what a wrong match looks like, so every constraint can be scored by the same cosine-similarity operator without sign inconsistencies. Ignorable constraints don't enter the listwise prompt at all; they act as a mask inside the deterministic rubric, preventing background or lighting cues from dominating scores.

Re-ranking runs in two stages. First, a deterministic rubric aggregates per-slot satisfaction and violation rates into a scalar score, with a configurable penalty for forbidden-constraint matches. Second, an MLLM listwise step re-orders only the top candidates (default five) using the evidence as an explicit anchor - keeping the online cost to two API calls per query regardless of candidate pool size. The same structured output - relevance scores on a 0-100 scale, a self-assessed confidence, and annotated hard pairs - doubles as distillation supervision for a lightweight student.

The numbers are substantial. On Flickr30k with BLIP-2, EviRank-pro reaches 95.61% R@1, beating the next-best CoTMR by 6.32 points. On COCO with CLIP-ViT-L/14 it hits 69.53% R@1, a 9.6-point gain over CoTRR. For fine-grained product retrieval on Stanford Online Products, it reaches 91.46% R@1, exceeding LoCoRE-base by 7.66 points. Even the rubric-only variant that uses no MLLM at test time outperforms all prior methods on those benchmarks.

Stability experiments are convincing. Across ten repeated calls, prompt paraphrases, and five different teacher models, Kendall's tau stays above 0.89 and R/F/I label agreement sits at 99% in all settings. Swapping Gemini-3-pro for the cheaper Gemini-3-flash costs 4.6 R@10 points but barely moves agreement rates - the fixed schema is doing the stabilizing work, not the teacher's raw capability.

The distilled student (Qwen3-VL-2B-Thinking, trained on 20k queries) retains over 90% of the teacher's capability and runs without any evidence generator at inference time. On Flickr30k it adds 10 R@1 points over CLIP coarse retrieval at 800ms per query.

The slot-semantics analysis is worth noting: pairwise SBERT similarity across the six slots averages 0.18, confirming they are not redundant restatements. The ablation showing that removing forbidden constraints hurts fine-grained discrimination (CUB-200 drops 1.96 R@1) while removing ignorable constraints hurts composed retrieval most (Toptee drops 1.04 R@10) validates that each constraint type is doing distinct work.