Ditch REINFORCE for a regression loss: agent-orchestration accuracy jumps +38 points in-distribution, +21 OOD

Notes on SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration (arXiv:2605.14089) — Mingda Zhang, Tiesunlong Shen, Haoran Luo, Wenjin Liu, Zikai Xiao, Erik Cambria, Xiaoying Tang · May 2026

Note published July 25, 2026 · written by SkillFed’s research pipeline from the paper above · how these notes are made

AI-assisted notes · reviewed by SkillFed Skill evolution

SkillFlow trains a single orchestrator (a "Supervisor" that calls a frozen executor and a growing library of reusable skills) with Tempered Trajectory Balance (TTB), a regression-style flow-matching loss borrowed from GFlowNets in place of a REINFORCE-style policy gradient. TTB pushes each trajectory's sampling probability toward being proportional to its reward, rather than collapsing onto one dominant path, so multiple distinct winning strategies survive under the same loss. Training TTB also produces a backward policy for free: it re-scores each action once execution feedback is known, giving per-step credit assignment at no extra inference cost. Two by-products fall out of this. A training-residual signal flags when the current skill library is capping performance, and a per-step flow signal shows where credit actually belongs. Together they drive an automatic curation loop that keeps, refines, or prunes skills, and mines new ones from matched success/failure trajectory pairs on the same query.

Across 14 benchmarks spanning QA, math, code generation, and interactive decision-making (7 in-distribution, 7 held out with the skill library frozen), a Qwen3.5-9B-based SkillFlow beat every baseline on every benchmark: direct prompting, supervised fine-tuning, GRPO, the search-based AFlow workflow method, and three RL/skill-evolution agents (AgentFlow, FlowSteer, SkillRL). Average gains over the untrained Qwen3.5-9B backbone ran +38.1 accuracy/pass-rate points in-distribution and +21.3 out-of-distribution, topping out at +61.7 points on WebShop success rate. Swap TTB back out for GRPO and the damage concentrates on diversity-sensitive tasks like AIME and WebShop, the tell that the gain is fixing mode collapse rather than just buying a bigger training budget. The full recursive skill-evolution loop also ran roughly 32-35% cheaper in tokens and wall-clock time than the SkillRL skill-evolution baseline.

Key numbers

Benchmarks evaluated	14 (7 in-distribution + 7 OOD)
Avg. accuracy/pass-rate gain, in-distribution	+38.1 pts vs. base Qwen3.5-9B
Avg. accuracy/pass-rate gain, out-of-distribution	+21.3 pts vs. base Qwen3.5-9B
Largest single-task gain	+61.7 pts, WebShop success rate
Token/time cost vs. SkillRL baseline	~32-35% lower

What this means for skill authors

For anyone building or maintaining a skill library, the transferable idea isn't the GFlowNet math. It's the discipline underneath it: a concrete signal for when to update a library (a training-residual plateau, not a fixed schedule), a ranking for what to keep or cut (measured marginal contribution to outcomes, not popularity or self-report), and a way to localize where a skill actually mattered inside a trajectory. Most skill catalogs today, including early-stage vetted ones, still lean on manual curation or an LLM asked to judge quality after the fact, which is exactly the unguided-evolution failure mode this paper targets. A coarser, non-GFlowNet version of the same instinct is a reasonable bar to hold any curated catalog to: track which skills correlate with successful task completion and prune the ones that don't, rather than trusting a skill's stated purpose.

The mode-collapse framing is the sharper point for skill users specifically. A REINFORCE-trained orchestrator that has converged on one winning path through a toolset has no fallback the moment that toolset changes; the paper calls this out directly as a brittleness once an underlying tool is upgraded. That's the normal condition for an agent pulling from a live, evolving catalog rather than a fixed one, where the right skill for a task can differ by which tools or backends happen to be available. It's the same reason SkillFed treats vetting as an ongoing scan rather than a one-time approval. An orchestration policy explicitly trained to keep multiple viable strategies alive, instead of collapsing onto whichever one scored best in training, fits that setting better than one optimized purely for benchmark accuracy.

Caveat: the code repository is still anonymized for review, so independent replication of these exact margins isn't yet possible, and the benchmark suite mixes standard datasets (HotpotQA, SWE-bench, ALFWorld) with results reported on a single Supervisor backbone family before generalizing to others. Treat the qualitative finding as the load-bearing claim, not the specific point gains: flow-matching beats REINFORCE for diversity-preserving skill orchestration, and a training signal beats heuristics for library curation.

Skills related to this research

playwright-verifier This skill automates accessibility compliance verification by testing whether your fixes actually resolve violations and don't create new ones. Built for Gemini, it provides structured testing workflows to catch accessibility regressions before they reach production.★ 370 statistical-analysis Conduct rigorous statistical tests—t-tests, ANOVA, chi-square, correlation, regression, and Bayesian methods—with systematic assumption verification and effect size reporting. The skill walks you through test selection, data inspection, assumption diagnostics, and APA-style write-ups so your analysis withstands peer review.★ 31,940 statsmodels Statsmodels is Python's statistical modeling library for fitting regression, generalized linear, time series, and discrete outcome models with full inference capabilities. It delivers detailed diagnostics, residuals analysis, hypothesis tests, and publication-ready output—ideal when coefficients and their uncertainty matter more than prediction alone.★ 31,940 agent-builder Agent Builder guides you through creating AI agents using the Claude Agent SDK and headless mode—no raw API keys needed. Build custom tools with MCP servers, control execution with hooks, and deploy agents in Python or via CLI commands. Covers SDK patterns, security practices, and production workflows.★ 274

Related notes

References

Mingda Zhang, Tiesunlong Shen, Haoran Luo, Wenjin Liu, Zikai Xiao, Erik Cambria, Xiaoying Tang. "SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration." arXiv:2605.14089 (2026).
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, Yoshua Bengio. "Flow Network Based Generative Models for Non-Iterative Diverse Candidate Generation." NeurIPS 2021.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, Yoshua Bengio. "Trajectory Balance: Improved Credit Assignment in GFlowNets." NeurIPS 2022.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, Emmanuel Bengio. "GFlowNet Foundations." Journal of Machine Learning Research 24(210), 2023.
Zhihong Shao et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300 (2024).