LLMs still can't match humans at reading between the lines of a biased retelling

Most benchmarks for social reasoning hand the model a complete, objective account of events and ask it to draw conclusions. Real advice-seeking looks nothing like that. A user comes to an assistant with a partial, emotionally colored retelling, and the assistant has to figure out what is actually going on from that alone. Fuse is a framework built to evaluate exactly this gap.

The mechanism is clever. A multi-agent simulation runs a social scenario — a new colleague, a roommate, a romantic interest — where one agent has a hidden motive. A separate user agent then recounts those events to the evaluated assistant, filtered through a controllable reporting bias. Because the ground truth is set at simulation time, no per-example human labeling is needed. Human annotators validated that the simulated behaviors faithfully manifest the intended motives in about 97% of cases, and that the first user message alone carries enough signal for a correct prediction in 88% of cases.

Across 12 models, none clears that 88% human-majority bar. The strongest model reaches 83.7 on the paper's MSR metric, leaving a gap of more than 6 points. That gap is not just about the inherent difficulty of social inference: the paper isolates a consistent additional cost from user mediation by comparing each model's performance when given the raw simulation events directly versus only the user's account. Even frontier models that nearly ace the observer task drop substantially in the assistant setting.

Two controlled axes make the framework diagnostically useful. Introducing a single-sentence opposing-belief prompt — nudging the user's framing toward the wrong motive — degrades all models, with an average drop of 7.7 MSR points against a human drop of only 3.6. That asymmetry suggests genuine susceptibility to framing rather than simple information loss. The detail-level axis shows that models often need more evidence than humans do to reach the same conclusion, and that their defaults can run in either direction: some escalate toward alarming readings even when the evidence is positive, others de-escalate when concern is warranted.

The multi-turn analysis is the most sobering part. Giving models up to eight turns to ask follow-up questions does help early on, but performance plateaus after turn four and sometimes degrades. When a model commits to a prediction at turn two, 81% of those positions hold through turn eight — and the shifts that do occur are as likely to be driven by accumulated user bias as by genuine new evidence. Longer conversations are not reliably better conversations.

The dataset of 21,600 examples is released as a static benchmark, which is a practical choice: because each evaluation instance is a fixed user message, any model can be evaluated without running the full simulation infrastructure.