SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback

Agent skills—portable modules encoding domain knowledge and handling procedures—have historically been static artifacts. They get authored once, and when they fail, the failure disappears into a closed ticket rather than feeding back into the skill itself. SkillEvo closes that loop, but its real contribution is more specific: it identifies why prior closed-loop approaches stall.

The diagnosis is precise. Single-turn QA evaluation can only see the gaps a user's opening message exposes. Patch those gaps, and the evaluation gradient decays—there is nothing left to find. TSR in the single-turn baseline climbs from 58.9 to 66.4 across four rounds, then plateaus. SkillEvo reaches 81.8 over the same four rounds. The ablation is clean: swap multi-turn interaction for single-turn QA while holding everything else constant, and the result collapses to exactly 66.4. The 15.4-point lead traces entirely to the feedback source.

The mechanism is a self-renewing gradient. Multi-turn simulation exposes defects in layers: once a shallow failure is patched, the dialogue can proceed further and surface the next layer. The case study makes this concrete—an agent stated a renewal rule exactly opposite to the correct one, gave a plausible follow-up answer that built on the false premise, and the user never flagged the contradiction. A single-turn evaluator would have missed it; sustained follow-up exposed it.

Simulation fidelity is addressed directly rather than assumed. An intent state machine tracks whether each key intent was raised and substantively addressed before allowing termination. Responsibility is split: if an intent was never raised, the failure is attributed to simulation distortion, not to the skill. Intent coverage reaches 98.9%, and blind expert comparison of 200 simulated dialogues against real tickets shows 95.3% agreement across intent expression, information-reveal pace, and emotion trajectory.

The governance layer handles a separate failure mode: successive revision degrades the skill's knowledge structure in ways a scalar pass/fail score cannot perceive—knowledge bloat, dangling references, factual over-generalization. Without governance, cumulative line-count growth reaches 16.2% by round four; with it, 2.8%. TSR improves by 51.8 points over the original skills while volume barely moves, which means the gains come from correcting existing knowledge rather than accumulating text.

The dataset is production tickets from Tencent Cloud customer support, all escalated to human agents—meaning every ticket represents a failure the current skill already cannot handle. The framework is deployed in that production environment, though the ticket data itself cannot be released due to privacy and confidentiality constraints. The method requires only multi-turn consultation logs with human reference solutions, so the approach generalizes beyond this specific source.