Three rounds of automated repair add up to 12pp of consistency — and beat a mature production system by 13.76pp

Notes on SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support (arXiv:2604.08618) — Xingyan Liu, Xiyue Luo, Linyu Li, Gang Huang, Jianfeng Liu, Hongli Qiao · April 2026 · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

SkillForge builds and maintains agent skills for cloud technical-support agents at Alibaba Cloud, closing a loop that most skill-authoring setups leave open. A Domain-Contextualized Skill Creator writes the initial skill by mining historical support tickets, tool-usage logs, and internal documentation — not a generic prompt working blind. From there, a three-stage pipeline (Failure Analyzer, Skill Diagnostician, Skill Optimizer) batches failed task executions, traces each one back to a specific section of the skill file, and rewrites only that section. The team ran it across five real cloud-support scenarios — Account, Domain, DNS, OSS, ECS — spanning 1,883 tickets and 3,737 tasks, with Qwen3-Max as the backbone and an LLM judge (validated at over 90% agreement with human experts) scoring each response against what a human engineer actually used to resolve the ticket.

Grounding skill creation in ticket history beats generating one generically, but the bigger lever is the self-evolution loop. Three rounds of diagnose-and-rewrite add up to roughly +9 to +12 points of strict consistency — no matter whether the skill started out generic, domain-mined, or hand-written by a human expert. Even the expert-authored skill, already the strongest starting point, still gains +10.99 points on its own — and after three rounds the evolved skill beats the mature, human-engineered production system it was measured against by +13.76 points. The gains taper, though: tool-use and tone errors keep shrinking round over round, but knowledge errors stop improving after round one. The authors trace that ceiling to contradictory historical answers, a deliberate policy against stuffing skills with case-specific facts, and tacit expertise that support staff never wrote down.

Key numbers

Evaluation scale	1,883 tickets / 3,737 tasks across 5 scenarios
Domain-grounded creator vs. generic creator	+4.3pp Strict CR / +3.6pp Lenient CR
Self-evolution gain after 3 rounds	+9.2 to +11.6pp Strict CR, any starting point
Evolved skill vs. production legacy system	+13.76pp Strict CR
Knowledge-failure reduction, round 1 to round 2	0% further reduction (plateau)

What this means for skill authors

For skill authors, this reframes the skill itself: a living artifact with its own feedback and repair cycle, not a one-shot prompt-engineering deliverable. The notable part is that an automated diagnose-and-rewrite loop out-improves even the expert who wrote the skill by hand — a systematic trace from failure to file section beats ad hoc human tuning, at least on the errors that surface as measurable inconsistency.

The more useful signal is where the automation stalls. Knowledge errors plateau while tool-use and tone errors keep improving, and the authors are upfront that no amount of iteration closes gaps rooted in contradictory source tickets or expertise nobody wrote down. That's exactly the defect a skill sitting untouched in a marketplace listing can't surface on its own; it just fails quietly in production. The case for checking skill quality after publication, not just certifying it once, gets stronger here — the same instinct behind scanning installed skills continuously instead of trusting a one-time listing review.

One caveat: the headline 4.3-point gap isn't a fully symmetric comparison. The generic-creator baseline was denied access to ticket content and internal knowledge bases under the same privacy rules that gave the domain-specific creator its edge, though both got the same mined tool schemas. The gap is real, but it measures the value of ticket-grounded knowledge extraction specifically — not generic skill creation writ large.

Skills related to this research

audio-router audio-router intelligently directs audio processing requests to the right specialized skill, whether for playback control, sound analysis, or reactive audio handling. Built for Claude, it streamlines how audio workflows are orchestrated across multiple focused capabilities.★ 8 seedance-recipes Seedance-recipes provides production-ready recipe patterns for video content across genres: product, lifestyle, drama, music video, landscape, commercial, animation, and more. Each recipe preserves core creative constraints while inviting customization of subject, camera, lighting, and sound. Use recipes as proven starting shapes, not rigid templates.★ 5,445 Socratic Questioning Sequence Generator This skill builds ordered question progressions designed to develop concepts through dialogue rather than direct instruction. It distinguishes genuine Socratic inquiry—which surfaces assumptions and builds understanding through student reasoning—from leading questions that funnel toward predetermined answers. The skill generates branching follow-up paths based on likely student responses, anticipating dialogue contingencies that teachers typically cannot construct in real time.★ 472 ugc-ads ugc-ads builds short-form product videos optimized for social platforms, combining jump-cut pacing with native lip-synced dialogue delivered as a first-person talking head. The skill auto-detects product category from your URL and structures the narrative across five beats—setup, name reveal, product reveal, twist, and punchline—then renders in your choice of vertical formats.★ 37

Related notes

References

SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support (arXiv:2604.08618)
Li et al., SkillsBench: benchmarking how well agent skills work across diverse tasks (arXiv:2602.12670)
Zhao et al., Agent-in-the-Loop: a data flywheel for continuous improvement in LLM-based customer support (EMNLP 2025 Industry Track, arXiv:2510.06674)
Zhang, Lazuka & Murag, Equipping agents for the real world with agent skills (Anthropic, Oct 2025)
Yao et al., ReAct: synergizing reasoning and acting in language models (arXiv:2210.03629)