Revisiting Sliced-Wasserstein Imitation Learning: What Survives on GPU PPO, and Where It Breaks
A technical post-mortem of SWIL from my PhD thesis—replacing adversarial max-GSW with exact 1D Kantorovich potentials in IsaacLab, testing it against official AMP on humanoid locomotion and Franka manipulation, and checking whether the same primitive transfers to RLVR post-training.
During my PhD, I worked on Sliced-Wasserstein Imitation Learning (SWIL)—the idea of replacing adversarial discriminators in distribution-matching imitation learning with fast 1D optimal transport along projections. Looking back at that early formulation with fresh eyes, it had two real flaws that kept it from being a clean method:
- Learned max-GSW projections reintroduce adversarial training. Maximizing over a neural projection is still a min-max game. Without strict Lipschitz/injectivity constraints it is unbounded and non-metric; once regularized, it behaves like a 1D WGAN discriminator with a quantile-sorting loss.
- Heuristic rewards do not match the first variation of . Using either an atom-replacement finite difference or a frozen squared residual does not give the exact policy gradient of , because the 1D monotone transport map itself depends on .
Recently I wanted a decisive answer: if we strip away all the adversarial heuristics, fix the math completely, and move to massively parallel on-policy GPU RL (IsaacLab), can SWIL be salvaged?
1. The Clean Fix: Exact-Dual SWIL on Fresh On-Policy GPU Batches
Fix unit directions , and let and be the 1D pushforwards of the policy and expert occupancy measures. For 1D quadratic optimal transport, the exact first variation of with respect to is the Kantorovich potential satisfying
On an on-policy GPU PPO rollout batch of transitions ( parallel envs steps), this is trivial to compute exactly:
- Standardize features using expert statistics (avoiding tight clipping that kills long-range transport gradients across disjoint supports).
- Project policy and expert batches to and , sort both on GPU (
torch.sort), and linearly interpolate sorted expert quantiles to form . - Integrate via cumulative trapezoid along the sorted grid, shift each slice by its minimum so , scatter back to sample order, and average .
On an RTX 5090, this takes 1.7–2.4 ms per PPO iteration for , , —with no discriminator network, no gradient penalty, and no off-policy replay staleness.
The terminating-MDP trap
In earlier off-policy MuJoCo tests (leanrl), exact-dual SWIL solved fixed-horizon HalfCheetah-v5 (10,238 return) while collapsing on all four early-terminating environments (Hopper, Walker2d, Ant, Humanoid). The reason is structural: is only unique up to an additive constant, and additive reward shifts are not policy-invariant in variable-horizon terminating MDPs (the classic DAC absorbing-state bias). Zero-centered assigns negative reward to half of transitions, incentivizing early termination whenever .
Mapping the shifted potential to a strictly positive bounded reward,
immediately eliminates early-termination suicide on IsaacLab humanoids—at the cost of turning quadratic 1D slice tail growth into a saturating Gaussian tail. (I also tested a polynomial-tailed Cauchy variant with top- slice weighting; it reduced foot skating slightly to 0.088 on Walk, but trailed plain exp_shift on SW2, speed, and survival.)
2. Where It Works—and Where Official AMP Still Wins
To avoid fooling myself with a weak baseline (my first custom LSGAN discriminator failed because a 20× GP scaling mismatch and large head init crippled it), I benchmarked a single frozen SWIL-Linear config (, , exp_shift, zero per-task tuning) directly against IsaacLab’s unmodified official skrl AMP trainer across all three built-in Humanoid-AMP tasks (Walk, Run, Dance; envs, iterations, evaluated deterministically over envs for steps):
| Task (ref speed) | Method | Ep Len ↑ | Tail p10 ↑ | fall < 150 ↓ | Speed (m/s) | Held-out SW2 ↓ | Foot Skate ↓ |
|---|---|---|---|---|---|---|---|
Walk (1.52 m/s) | Official skrl AMP | 291.0 | 299.0 | 3.1% | 1.17 | 1.353 | 0.103 |
Walk (1.52 m/s) | SWIL-Linear (frozen) | 268.7 | 43.5 | 11.3% | 1.26 | 0.876 | 0.113 |
Run (3.41 m/s) | Official skrl AMP | 240.0 | 28.0 | 21.9% | 2.16 | 1.459 | 0.171 |
Run (3.41 m/s) | SWIL-Linear (frozen) | 211.7 | 21.0 | 32.4% | 1.05 | 1.321 | 0.104 |
Dance (0.48 m/s) | Official skrl AMP | 261.9 | 43.5 | 14.5% | 0.39 | 1.694 | 0.179 |
Dance (0.48 m/s) | SWIL-Linear (frozen) | 62.6 | 37.0 | 99.6% | 0.66 | 1.263 | 0.267 |
On steady periodic Walk, SWIL-Linear produces clean, upright bipedal locomotion out of the box—matching official skrl AMP on foot skating (0.113 vs 0.103), tracking reference speed closer (1.26 vs 1.17 m/s), and cutting held-out SW2 by 35% (0.876 vs 1.353) with zero discriminator. It also cleanly beat non-adversarial RBF-MMD and random-Fourier-feature (swil_rff) baselines, both of which collapsed to flat +0.5 alive bonuses (rew_std = 0.034).

Side-by-side 240-step PhysX rollout comparison on Humanoid-Walk (green: reference mocap; blue: SWIL-Linear; orange: official skrl AMP). Download the animated clip: walk_compare_skeleton.mp4.
Yet the 3-task table exposes the exact ceiling of fixed-slice OT:
SWIL-Linearachieved lower held-outSW2than officialskrlAMP on every single task (Walk,Run, andDance). The optimizer did its job.- Even so, official
skrlAMP beatSWIL-Linearon closed-loop survival and behavioral fidelity on every task, and the gap widened monotonically with motion complexity: small on periodicWalk, noticeable on high-speedRun(1.05vs2.16 m/s), and catastrophic on non-periodic agileDance(ep_len = 62.6,99.6%falling before step 150).
3. The Sharpest Falsification: Precision Contact Manipulation
To test the fixed-horizon pure-IL regime (removing termination bias completely), I trained an oracle PPO policy on Isaac-Lift-Cube-Franka-v0 (97.3% train success, 90.0% deterministic demo success across 50 saved trajectories) and ran pure IL with zero task reward:
Method (Isaac-Lift-Cube-Franka-v0, 50 demos) | Final Lift Success ↑ | Peak Success | Task Return ↑ | Held-out SW2 ↓ | Rew ms/iter |
|---|---|---|---|---|---|
| Oracle PPO (shaped task reward) | 0.973 (demo 0.900) | 0.980 | 107.18 | — | 0.0 |
AMP / GAIL (LSGAN+GP) | 0.094 | 0.191 | 2.61 | 0.450 | 17.4 |
RBF-MMD | 0.004 | 0.008 | 0.94 | 0.327 | 17.8 |
SWIL-Linear | 0.000 | 0.008 | 1.04 | 0.362 | 3.5 |
SWIL-RFF | 0.000 | 0.004 | 0.88 | 0.378 | 3.5 |
PWIL | 0.000 | 0.000 | 0.70 | 0.592 | 0.6 |
Both SWIL-Linear and RBF-MMD drove marginal SW2 down to 0.33–0.36 while achieving 0% lift success. In the 44-dimensional (obs, act) vector, grasping and lifting depend on a tiny nonlinear contact subspace (3 cube coordinates coupled with gripper closure). Isotropic random slices weight all 44 coordinates equally, so the policy matches ~40 arm-kinematics coordinates and ignores the cube. A learned discriminator—despite higher marginal SW2 (0.450)—focuses capacity directly on the discriminating contact subspace.
4. Does Exact-Dual Sliced OT Transfer to RLVR Post-Training?
Before closing the book on the exact-dual primitive, I ran one more falsification test in a completely different regime: diversity regularization in RL with verifiable rewards (RLVR). The hypothesis was that a correctness-gated exact-dual bonus in a frozen embedding space (matching correct policy rollouts against a reference base-model distribution) could preserve solution diversity and improve held-out pass@16 over plain GRPO+KL without collapsing pass@1.
I tested three matched-budget arms on Qwen/Qwen2.5-1.5B-Instruct trained with LoRA GRPO on MATH (Levels 3–5, 300 steps, kl_beta = 0.005, lr = 5e-5, lambda_div = 0.25) and evaluated on n = 256 held-out MATH-500 prompts (k = 16 samples/prompt, temperature 0.8):
Arm (300 steps, MATH-500, n = 256) | pass@1 ↑ | pass@4 ↑ | pass@8 ↑ | pass@16 ↑ | Correct Emb. eff_rank | Total Correct (/4096) |
|---|---|---|---|---|---|---|
GRPO + KL | 0.3660 | 0.5199 | 0.5852 | 0.6484 | 43.65 | 1,499 |
GRPO + cosine repulsion | 0.3486 | 0.4912 | 0.5505 | 0.6094 | 43.20 | 1,428 |
GRPO + SW exact-dual | 0.3406 | 0.4945 | 0.5607 | 0.6172 | 43.36 | 1,395 |
Paired bootstrap 95% confidence intervals (10,000 resamples over the 256 matched prompts) confirm a clean negative result:
sw_exact_dual − kl:−2.54 pponpass@1([-3.98, -1.12] pp),−2.54 pponpass@4([-4.62, -0.42] pp), and−3.12 pponpass@16([-7.81, +1.17] pp).cosine − kl:−1.73 pponpass@1([-3.15, -0.32] pp) and−3.91 pponpass@16([-7.81, 0.00] pp).sw_exact_dual − cosine: statistically indistinguishable across all .
Both embedding-space diversity bonuses distorted within-group GRPO advantages—rewarding surface embedding spread among correct rollouts rather than reliable reasoning—while adding ~25–28% wall-clock overhead and producing no gain in effective rank (43.36 vs 43.65).
5. Closing the Chapter
Together, the robotics and RLVR experiments give a clean, falsifiable verdict on SWIL and close this thread from my thesis:
- What works: Exact-dual 1D Kantorovich potentials on fresh on-policy GPU batches are genuinely fast (
~1.7 ms/iter), stable, and easy to deploy—capable of learning upright periodic bipedal locomotion without an adversarial discriminator. - Why it is not a general AMP replacement: Minimizing marginal linear does not guarantee closed-loop physical competence when success depends on nonlinear dynamical couplings (
Dance, high-speedRun), sparse contact subspaces (Franka Lift), or support containment rather than mass conservation. - Why it did not transfer to RLVR: In verifier-driven post-training, adding an embedding-space sliced-OT bonus on top of binary correctness rewards degraded
pass@1throughpass@16relative to plainGRPO+KL. - Where the primitive still belongs: Only when global distribution matching in a compact, dense, task-aligned feature space is literally the objective (e.g., matching unlabeled telemetry distributions for sim-to-real calibration)—not as a drop-in proxy for contact physics or reasoning diversity.