Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

The core problem Co-RL addresses is simple to state but hard to escape: when a model generates its own training signal, it tends to reinforce whatever it already believes. Self-rewarding RL methods like TTRL derive pseudo-labels from majority votes over the model's own rollouts, which means a model that is confidently wrong gets rewarded for being confidently wrong. The paper demonstrates this failure mode empirically — prolonged TTRL training produces reward collapse, degenerate completion lengths, and accuracy that peaks and then degrades.

The fix is to break the feedback loop by having each model scored against a different model's majority vote. Co-RL trains multiple agents simultaneously, each generating rollouts on the same unlabeled prompts, but each receiving rewards based on a peer's pseudo-label rather than its own. No parameters are shared, no gradients cross between agents, and no external judge or learned reward model is required. The reward signal is just: does your answer match what your partner's majority vote says?

The theoretical analysis makes the mechanism precise. Under self-rewarding GRPO, the update direction is determined by the same rollout group being optimized, so the dynamics are self-confirming — whichever answer is currently more probable gets amplified regardless of correctness. Under Co-RL, the update direction is determined by the peer's signal, which is independent of the agent's own rollout. The paper proves that this expands the basin of attraction for the correct answer: Co-RL converges correctly whenever the joint initialization satisfies a specific threshold condition, whereas self-rewarding converges correctly only when a single agent already exceeds that threshold alone.

Diversity is not incidental — it is the mechanism. The paper measures error overlap across model pairs before any training, finding that different-family pairs (e.g., Qwen2.5 paired with Llama-3) show substantially lower wrong-agreement and higher complementarity than same-family pairs, which in turn are indistinguishable from pairing a model with a different random seed of itself. The practical implication: same-family cohorts stabilize training but provide limited corrective signal on roughly three-quarters of problems; different-family cohorts are the cheapest way to buy genuine independence.

The empirical results hold across both text and multimodal settings. On seven text-only benchmarks, Co-RL improves four LLMs by 3.0–8.6% on average over their base models, outperforming the strongest self-rewarding baseline by 0.8–2.0%. On four multimodal math benchmarks, five VLMs ranging from 2B to 12B parameters improve by 2.3–7.2%. In the controlled CoMAS multi-agent comparison, Co-RL outperforms prior multi-agent RL methods by 4.0% on average while using half as many agents and requiring no LLM judge. In several settings — notably Llama-3.1-8B-Instruct and Gemma-3-12B — Co-RL without any ground-truth labels surpasses GRPO trained with verified answers.

The matched-budget ablation is worth noting: training two models independently with TTRL and ensembling their outputs at inference provides only marginal gains over the stronger individual model. Co-RL ensembles better because the agents were trained to have complementary errors, not just independent ones. The gains come from the training interaction, not the inference aggregation.