Accuracy is a gate, not a dial: why valid agentic data matters more than volume

The central argument here is that more agentic training data is not the same as better agentic training data. This survey introduces a three-part framework — Accuracy, Complexity, divErsity (ACE) — and applies it to the sprawling literature on generating interaction data for LLM agents. The framing is deliberately asymmetric: accuracy is an admission gate, not a dial. An environment whose tools are inconsistent, a task that is infeasible in its stated environment, or a trajectory whose observations don't follow from executed actions cannot be rescued by being difficult or varied. Complexity and diversity only matter inside the valid set.

The factorization underlying ACE is compact but does real work. Every agentic data sample is decomposed into an environment specification, a task signal, an interaction realization, and an optional verifier. This four-part object accommodates everything from short function-calling examples to long stateful coding trajectories without forcing them into the same serialization format. More usefully, it makes the targets of each ACE dimension explicit: accuracy concerns consistency among the four factors, complexity concerns the burden their configuration places on a declared learner, and diversity concerns non-redundant coverage across all three.

The complexity treatment is the sharpest section. The paper argues that horizon, tool count, and graph depth are weak proxies — an explicit dependency chain can actually make a task easier by revealing the solution path, and an added constraint can prune the search space rather than expand it. What matters is whether structural choices lower verified success rates for a specific model under a specific scaffold and inference budget. The useful complexity band is also a moving target: as the learner improves, previously hard tasks saturate and the frontier shifts, which means static generation eventually stops contributing.

On diversity, the paper is equally skeptical of surface metrics. Renamed tools or paraphrased requests that induce identical action sequences are not diverse in any meaningful sense. The paper proposes measuring behavioral non-redundancy through tool-call graphs, state transitions, and dependency paths rather than text similarity or dataset size. One finding worth noting: a smaller set of robustly verified, behaviorally distinct environments can outperform a larger but redundant collection — a direct challenge to the assumption that scaling data volume is the primary lever.

The survey also traces a clear directional shift in the field: from plausibility-based filtering toward execution-grounded verification, from static difficulty heuristics toward model-relative calibration, and from surface variation toward behavioral coverage. The discussion of self-evolving agents extends this into a closed-loop framing where data generation becomes a continual process of identifying capability gaps and allocating verified experience — not a one-time production step.

The paper is a survey, so its contribution is organizational rather than empirical. It does not introduce new training results. But the ACE lens is genuinely clarifying for anyone designing a data pipeline: it separates what a pipeline constructs from what the resulting distribution should look like, and it gives precise language for tradeoffs that practitioners currently navigate by intuition.