Noisy labels in RGB-D benchmarks have been hiding real model performance for years

Annotation quality in RGB-D benchmarks has been quietly sabotaging model evaluation for years. This paper makes that case with a concrete number: re-evaluating a pretrained model on a cleaned version of SUN RGB-D's test set alone lifts mIoU from 52.8 to 55.0 — a meaningful jump that reflects measurement error, not model improvement. When the training data is also cleaned, the same model reaches 59.0. The gap between what researchers thought their models could do and what they actually can do is partly a labeling artifact.

The dataset itself, RGBD20K, is a direct response to that problem. Twenty thousand RGB-D image pairs, 160 fine-grained categories, 75 scene types — compared to NYUv2's 1,449 pairs and 40 classes, or SUN RGB-D's 10,335 pairs and 37 classes, the scale difference is substantial. The data was assembled from several existing sources — SUN RGB-D, tracking benchmarks, the DIML RGB-D dataset, and others — then unified under a single taxonomy through multi-stage manual refinement. That curation work is the real contribution; the paper is explicit that the annotation noise in prior datasets isn't a minor inconvenience but a structural problem that distorts both training and evaluation.

The accompanying Score-Purified Fusion (SPF) model is presented as a baseline rather than the headline. Its design logic is straightforward: before cross-modal attention runs, it computes reliability scores for features from each modality and uses those scores to filter the Key and Value representations. The idea is that standard cross-attention gets diluted when it has to simultaneously resolve sensor noise and depth boundary misalignment while also aggregating context. Purifying first lets attention focus on the cleaner signal. The architecture builds on GeminiFusion with a SwinTransformer Large backbone, trained on four H100 GPUs. Ablations confirm that bidirectional score enhancement — applied to both RGB and depth branches, and to both Key and Value — outperforms any partial version.

The dataset's long-tail distribution is intentional, mirroring real-world object frequencies rather than artificially balancing classes. Object parts are annotated and linked to parent objects, so a drawer is connected to its cabinet. Depth-aware ordering handles occlusions by assigning relative depth layers from the actual depth map rather than guessing from image geometry alone.

For anyone building perception systems that need to generalize beyond the narrow indoor corridors that NYUv2 and SUN RGB-D were built from, the category and scene diversity here matters. The annotation quality argument also has implications beyond RGB-D: if cleaning SUN RGB-D's labels shifts mIoU by more than two points on a frozen model, other benchmarks in adjacent domains likely have similar hidden floors.