All notes

Treating the Training Distribution as an Optimization Variable in RL

Predictive Safety Curricula and the case for choosing which rollouts to generate under a fixed interaction budget.


Most progress in reinforcement learning is framed around better policy optimizers, larger architectures, or sharper reward functions. Far less attention goes to an equally decisive lever: the training distribution itself—which states, perturbations, and task configurations a policy actually encounters under a fixed interaction budget. In practice, training distributions are still treated mostly as static infrastructure: a fixed set of environments, a uniform domain-randomization box, or a hand-tuned difficulty ladder driven by scalar task completion.

That framing breaks down once policies become broadly competent. When an agent solves ninety-plus percent of nominal trials, average reward saturates and uniform sampling becomes deeply inefficient. Most rollouts confirm behaviors the policy already mastered, while rare, high-consequence failures remain starved of gradient signal. Worse, coarse performance metrics are often blind to the difference between clean execution and brittle execution. Two stair configurations in legged locomotion can yield identical traversal rates while differing sharply in how often the robot clips a step with its lower leg. A curriculum keyed solely to task completion or learning progress treats both regimes as solved.

In Predictive Safety Curricula (PSC) (arXiv:2609.37070), we tested a direct alternative: keeping the policy objective, architecture, and total rollout budget fixed, while turning experience allocation into a closed-loop variable. Instead of relying on task progress alone, a learned distributional safety critic predicts future safety cost from policy rollouts and reallocates experience inside the current capability support—shifting both context sampling and event replay toward conditions with elevated anticipated risk. On rough-terrain simulation, reallocating the same interaction budget reduced failure rates by 22.3% relative to a learning-progress curriculum; on ANYmal-D hardware, it cut shank-collision incidence by 63% across matched seeds and eliminated collisions across 50 production stair trials without slowing traversal.

Although validated on physical robotics, the underlying bottleneck is not specific to locomotion. The exact same pathology appears across modern RL—from continuous control and manipulation to agentic post-training and tool-use benchmarks. Whenever rollouts are expensive and failures live in the long tail, static prompt mixtures and pass-rate filtering systematically misallocate compute. They oversample saturated tasks while under-exposing the policy to edge cases: brittle multi-step recoveries, subtle constraint violations, or off-distribution tool errors that rarely trigger under naive sampling. Treating auxiliary critics, verifiers, or uncertainty estimators as allocation signals—rather than merely post-hoc filters or reward-shaping terms—separates what the policy should optimize from where it should spend its next thousand rollouts.

Reweighting an existing support of contexts and replaying previously observed perturbations is only the first step. The deeper research problem is sequential experimental design for RL: moving from passive reweighting to active task selection and environment generation. Rather than sampling from a predefined pool, an outer loop can maintain explicit estimates of policy uncertainty and anticipated failure severity, then actively choose—or synthesize—the next environment parameter, prompt structure, or adversarial perturbation expected to yield the highest reliability gain per unit of compute. In this view, curriculum learning, red-teaming, and automated evaluation collapse into a single loop: every evaluation run informs not just a leaderboard number, but the next training distribution.

Making this paradigm work reliably at scale exposes three sharp open problems:

  • Non-stationarity and critic lag. As the policy improves, yesterday’s failure modes vanish and new blind spots appear. If the predictive risk or uncertainty model updates too slowly—or overfits to stale replay events—allocation chases obsolete failures and wastes rollouts.
  • Tail focus versus capability preservation. Over-concentrating experience on rare failure regimes can induce distribution collapse, degrading nominal throughput or causing forgetting on common cases. Cleanly decoupling capability expansion from risk-weighted allocation remains difficult when the two interact tightly.
  • Designing the acquisition objective. Pure epistemic uncertainty favors unsolvable noise; pure predicted cost favors narrow, repetitive hazards. Balancing learnability, epistemic uncertainty, and downstream consequence—without creating a new specification-gaming surface inside the curriculum generator itself—is still largely unsolved.

Scaling RL past average-case competence requires treating the data-generating process with the same algorithmic rigor we apply to policy updates and reward modeling. The central question is no longer only how a policy should learn from a batch of rollouts, but which rollouts are worth generating next.

Reinforcement LearningCurriculum LearningActive Experimental Design