Mobile agents improve faster when the harness evolves alongside the model

The central claim here is architectural: you cannot improve a deployed agent by updating only its model weights, because the scaffold around the model—the harness supplying tools, memory, and procedural guidance—shapes what the model actually does. Qwen-Planner-Agent is built around that premise, and the paper is largely an account of what it takes to operationalize it.

The system has three interlocking development loops. The data loop uses AI-assisted task construction and failure diagnosis to generate training tasks, then adjusts sampling weights as the model masters or struggles with different capabilities—not just adding more examples, but retiring redundant ones and constructing new tasks for uncovered gaps. The training loop starts with supervised fine-tuning on curated trajectories, then moves to online reinforcement learning across three environment types: deterministic programmatic sandboxes, LLM-simulated long-tail interactions, and selective real-device sessions. The third loop is the one most papers skip: iterative revision of the harness itself, alternating with model training so that each informs the other.

The RL contribution, CARE (Competence-Aware Reward-and-Advantage Engineering), addresses a specific failure mode in group-based policy optimization. Once a task group reaches high success rates, standard within-group normalization amplifies small efficiency differences to roughly the same advantage scale as groups with genuinely mixed outcomes. CARE imposes a success-derived floor on the normalization denominator, keeping efficiency signals proportional to their actual weight in the reward. The ablation shows this matters: removing advantage calibration produces shorter outputs but substantially lower accuracy over the training interval, while full CARE reduces output length by about a third at the final measured step without sacrificing accuracy relative to vanilla RL.

On MobilePA-Bench—over 1,700 executable tasks across 13 domains—the 27B system scores 77.05% overall, edging out GPT-6 Astra at 76.84% and Claude Opus 5 at 75.71%. The estimated output cost is $2.41 per thousand tasks against a range of roughly $3 to $68 for the comparison models, though the paper is careful to note this covers output tokens only, excluding input tokens, tool charges, and device execution.

The memory evaluation is the most concrete demonstration of what the harness actually buys. On the BEAM benchmark at the 10M-token history scale, direct-context models top out around 25.64% under truncation; adding the harness's persistent memory manager raises the matched Qwen-Planner-Model 27B score to 67.24%. At shorter histories the gains are negligible or occasionally negative, which is the honest result: structured memory retrieval only outperforms direct context when context genuinely cannot hold the relevant history.

Model–harness co-evolution over four iterations improves MobilePA-Internal from 82.67% to 88.50% and MCPMark from 38.00% to 46.98%, both exceeding the harness-only baseline. The paper frames this as a development pathway rather than a claim of autonomous co-evolution—human review gates every data release and harness revision, and model parameters stay fixed during serving. That restraint is appropriate given the scope of the experiments, and the paper is transparent about it.