Splitting advantages by segment is a simple, honest fix for tool-calling RL

Standard GRPO assigns a single trajectory-level advantage to every token in a rollout. For tool-calling agents, whose outputs split cleanly into an execution segment (reasoning plus structured API calls) and a summary segment (the user-facing response), this creates a specific failure mode: a fluent, correct summary can inflate the advantage for a broken or redundant tool trajectory, while a poor summary can suppress credit for perfectly executed tool calls. The paper names this Cross-Segment Credit Misattribution and demonstrates it concretely—a diagnostic case shows an agent issuing unnecessary sequential calls instead of a single parallel invocation, yet receiving a reinforcing gradient because the summary reward dominates the unified advantage.

The fix, Segment-Locked Credit Assignment (SLCA), is mechanically simple. Tool tokens and summary tokens are identified deterministically from the existing environment-boundary mask—no learned segmenter required. Two separate reward signals are normalized independently within the rollout group, then routed exclusively to their corresponding token segments before the backward pass. Summary rewards never touch tool-token gradients; tool rewards never touch summary-token gradients. The paper proves this is an advantage-level firewall rather than a gradient-direction firewall: measured cosine similarity between tool and summary gradients stays near zero throughout training, so the contamination shows up in advantage magnitudes, not directions.

The theoretical framing is careful about what it claims. The variance-removal result is conditional and local—it isolates one noise channel that unified broadcasting necessarily exposes, without asserting unconditional variance dominance. The authors frame SLCA explicitly as a bias–variance trade-off: routing removes summary-noise variance from tool updates but introduces a grouping bias on the summary side, since rollouts are normalized across trajectories that may have diverged after the tool segment.

Empirical results across three backbones (Qwen2.5-3B, 7B, and Qwen3-8B) are consistent. On the 7B backbone, SLCA-GRPO beats matched GRPO by 2.53 percentage points on in-domain Toucan, 1.36 pp on BFCL, and 9.15 pp on τ-Bench. The τ-Bench gaps are larger at 8B (+10.03 pp), which involves multi-turn collaborative agent tasks where execution errors compound. Gradient-norm volatility is lower under SLCA across all three scales, with the largest reduction at 3B (25%) and the smallest at 8B (2.5%).

The paper is honest about what SLCA does not solve. It routes one scalar advantage to every tool token, so it cannot distinguish a correct first action from a failing second action within the tool segment—that temporal credit problem is orthogonal and explicitly left for future combination with methods like VinePPO. The segment decomposition also assumes a clean boundary between tool calls and free-form text, which breaks for settings like inline code generation. And experiments top out at 8B parameters with a simulated tool environment; real-API behavior at larger scale remains untested.