Sampling guidance depth from a Gaussian beats searching for a single optimal value

Hint-based reinforcement learning for long-horizon agents works by replaying a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success rather than from scratch. The critical variable is how much of that trajectory to keep. Too little and the policy never reaches rewarding states; too much and every rollout succeeds, leaving nothing to learn from. Existing methods treat this depth as a single number — either one value shared across all tasks in a batch, or one value estimated per task through extra probe rollouts.

Agent-G² starts from a diagnostic observation: the set of depths that produce useful training signal is not a point but a band, and the training-signal profile across that band is unimodal, approximately symmetric, and well described by a Gaussian. Shared-depth schedulers place over 60% of assignments outside this informative band. Per-sample probing reduces mismatch only by spending rollouts that remain too noisy under a realistic budget.

The fix is to stop searching for a single optimal depth and instead sample from the band. Agent-G² maintains a global baseline tracking overall policy progress, plus per-cluster statistics — clusters defined offline by expert-trajectory length as a proxy for task difficulty. Each cluster tracks an exponential moving average of success rate and its variance. The per-task Gaussian center shifts toward deeper guidance when a cluster's success rate is low; the spread widens when within-cluster variance is high. One depth is drawn per task from that Gaussian, and the same rollouts that update the policy also refresh the Gaussian parameters. No probe rollouts, no auxiliary network.

On ALFWorld with Qwen2.5-1.5B, Agent-G² outperforms the strongest hint-based baseline by 2.3 points and the strongest hint-free baseline by 7.4 points, at under one-third the rollout cost of per-sample probing. The 1.5B model also surpasses all 7B non-probing baselines, suggesting that schedule design can substitute for backbone scaling rather than merely compensating for limited capacity.

Ablations confirm that each component earns its keep. Replacing the Gaussian draw with its deterministic mean drops overall success noticeably, showing that stochastic coverage of the band matters, not just locating its center. Replacing the Gaussian with a variance-matched uniform distribution also hurts, confirming the Gaussian shape contributes beyond mere coverage. Removing cluster-aware centering or adaptive spread both degrade performance, with the largest drops on Long-horizon tasks.

The main stated limitation is that the framework requires one expert trajectory per training task. When demonstrations are unavailable or expensive, Agent-G² cannot be applied directly. The cluster partitioning is also fixed offline by trajectory length, which does not adapt as the policy improves and relative task difficulty shifts.