One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

Pooled reinforcement learning on software engineering tasks hides a structural problem: gains in one task category routinely coincide with regressions in another, while the aggregate resolution score stays flat or rises. This paper names that pattern the category see-saw and builds an entire training framework around diagnosing and correcting it.

The core observation is empirical. Training a single policy on a 6,723-task mixture of repository-level SWE tasks produces opposing category-level movements throughout training. A balanced subset—516 tasks per category, 1,548 total—raises the average minimum category gain slightly and matches the pooled endpoint's overall resolution (55.34% versus 55.50%), but does not eliminate the opposing movements. Balancing the mixture is not enough.

The proposed alternative separates training entirely. Three category experts—service/data/security (A), user-facing applications (B), and systems/tooling/runtimes (C)—each start from the same Qwen3.6-27B base and train on their own task pools. The categories come from SWE Labeler, a hierarchical labeling system with 26 Task Type L1 families, 119 L2 labels, 21 Repository Domain L1 families, and 108 L2 labels, each grounded in sources like ISO/IEC 25010, MITRE CWE, and Fowler's refactoring catalog. Repository Domain L1 does the routing; the other axes support analysis.

Initial category-specific RL improves average training success rates but leaves roughly 30% of training instances with lower observed pass rates than before. This motivates Refresh–Repair–Expand (RRE): after each RL phase, the expert refreshes its mastery estimates, replays its own verifier-approved successful trajectories toward low-mastery instances via Repair SFT, then expands its training frontier by probing previously excluded tasks. No external model supplies trajectories at any stage. After two full RRE rounds, the three experts exceed the base model on their target categories by 7.84, 4.48, and 5.61 percentage points respectively, and all three surpass the Pooled RL endpoint on their respective categories.

Consolidating three experts into one deployed model uses label-routed multi-teacher on-policy distillation (MOPD). The student generates its own trajectories; each task is routed to the corresponding expert teacher, whose log-probability advantage guides the student update. A ReLU-gated extrapolation term adds a reference-anchored direction on tokens where the teacher already exceeds the reference, keeping only the improving direction. The student's training rollout score rises from roughly 45.5% to 56.9% during integration, while student–teacher KL falls by 43%.

The final MOPD policy reaches 58.04% mean resolution on Pro-618 and 59.00% on SWE-bench Multilingual—gains of 5.39 and 2.78 percentage points over the base. It improves over Pooled RL in every category, with minimum category lifts of 1.53 points on Pro-618 and 3.22 points on Multilingual. Expert-gain recovery is uneven: 80.8% on A, 111.1% on B (the student exceeds its teacher), and 84.8% on C. The authors flag A-category retention as the clearest remaining gap.

The limitation the paper acknowledges most directly is that hard single-category routing simplifies overlapping task structure, and the experiments evaluate the complete framework rather than isolating individual components.