SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

Task success is a lie of omission. SoftVTBench makes that concrete: across every one of twelve in-distribution policy configurations tested on deformable objects, some fraction of successful rollouts exceeded the calibrated deformation tolerance. For Diffusion Policy's visuo-tactile continuous variant, that fraction reached roughly half of all successes on both deformable suites. A completion-only protocol would have accepted every one of those episodes.

The benchmark's central mechanism is a deliberate split between what the policy sees and what the evaluator scores. During a rollout, the policy receives RGB, tactile images, marker-motion fields, proprioception, and language. The evaluator reads finite-element nodal positions that are never exposed to the policy, computes rigid-motion-removed RMS displacement normalized by bounding-box diagonal, and checks it against a per-object tolerance fixed before any policy was trained. That tolerance comes from a scripted calibration sweep—the 90th percentile of peak displacement over stable grasps—so no evaluated policy can influence the criterion against which it is scored. The Deformation-aware Success Rate (DSR) credits an episode only when it both completes the task and stays within that tolerance. The gap between TSR and DSR is exactly the count of episodes a completion-only protocol accepts that SoftVTBench rejects.

Three findings from the experiments are worth dwelling on. First, the gap is policy-dependent, not a fixed cost of handling soft objects. FastWAM keeps TSR and DSR within a percentage point on spatial configurations while achieving the strongest spatial results overall—an existence proof that the gap is a tractable problem rather than an unavoidable one. Second, apparent tactile gains can be entirely explained by gripper control granularity. For the policy family where the ablation was run, switching from binary to continuous control alone raised task success by roughly as much as adding touch did; combining both yielded no improvement over continuous control by itself. Without storing both gripper-action encodings in the same demonstrations—which SoftVTBench does—that confound is invisible. Third, touch is more reliably associated with robustness under distribution shift than with peak in-distribution performance. Visuo-tactile variants beat vision-only in task success across all six out-of-distribution policy–suite comparisons and in DSR in five of six; in distribution the same comparison is roughly split. The value of contact sensing reads as resilience, not peak performance.

The dataset itself pairs 4,000 expert demonstrations across four diagnostic suites—two deformable, two matched rigid twins—with ten volumetric soft assets and synchronized tactile rendering via simulated GelSight Mini sensors. The paper is explicit that no real-sensor comparison was performed and that sim-to-real transfer of the tactile signals is unvalidated; the conclusions are stated for simulated policy learning only.

What SoftVTBench actually delivers is a scoring infrastructure that separates whether a policy succeeded from how it physically behaved while succeeding. That distinction has been missing from every prior benchmark in this space, and the experiments show it changes conclusions that completion alone would support.