Auto-evolved skill specs beat hand-written ones by 7 points, at twice the constraint density

Notes on SkillChain: Closing the Loop on Skill Evolution for Image-Based E-Commerce AI Assistants (arXiv:2606.12984) — Yimin Hu, Mengtao Xu, Haoze Guo, Yuheng Song, Xiaoyong Zhu, Bo Zheng · June 2026

Note published July 23, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

E-commerce visual assistants take one uploaded photo and have to route it to wildly different behaviors — product search, style advice, an encyclopedia lookup, a plain utility call — and each one needs its own response shape, tool calls, and domain constraints. SkillChain takes the hand-authored behavioral spec for each intent (a "Skill") and makes it evolve on its own, through three linked stages. A Skill Creator bootstraps an initial Skill Bank from task specs and real user trajectories behind a human reflection gate. A Route Optimizer mines live routing failures — ambiguous intent boundaries, missing skills, parsing errors — and updates, merges, or discards Skill descriptions, with a provable guarantee that routing F1 never regresses round over round. A Body Refiner closes the loop on content quality: a dual-path evaluator pairs rule-based structural checks with an LLM-as-judge that scores every response on four axes — tool-call rationality, card-composition compliance, content quality, constraint adherence — then aggregates that signal across many queries before rewriting the Skill body.

Deployed against a production-scale e-commerce image assistant and measured on 1,000 intent-stratified queries, the full pipeline lifts the aggregate LLM-judge score from 59.1 (no skill at all) to 72.2, beating even a hand-written ManualSkill baseline (64.9) — with the sharpest jump on card-composition compliance, from 49.1 to 72.2. An ablation shows the aggregation step matters more than it looks: dropping cross-query statistical aggregation alone costs 12.2 points on that same card-compliance score, more than removing the LLM judge itself. Self-evolved Skills also end up roughly twice as long as manually written ones (150–200 constraint lines versus 60–80), and a one-week online A/B test confirms the offline gains carry over to real users — a 4.98-point full-read-rate gain and a 1.15-point bump in 7-day return rate.