All notes

Revisiting Sliced-Wasserstein Imitation Learning: What Survives on GPU PPO, and Where It Breaks

A technical post-mortem of SWIL from my PhD thesis—replacing adversarial max-GSW with exact 1D Kantorovich potentials in IsaacLab, testing it against official AMP on humanoid locomotion and Franka manipulation, and checking whether the same primitive transfers to RLVR post-training.


During my PhD, I worked on Sliced-Wasserstein Imitation Learning (SWIL)—the idea of replacing adversarial discriminators in distribution-matching imitation learning with fast 1D optimal transport along projections. Looking back at that early formulation with fresh eyes, it had two real flaws that kept it from being a clean method:

  1. Learned max-GSW projections reintroduce adversarial training. Maximizing W2((gψ)#ρπ,(gψ)#ρE)W_2((g_\psi)_\#\rho_\pi, (g_\psi)_\#\rho_E) over a neural projection gψ:X→Rg_\psi : \mathcal{X} \to \mathbb{R} is still a min-max game. Without strict Lipschitz/injectivity constraints it is unbounded and non-metric; once regularized, it behaves like a 1D WGAN discriminator with a quantile-sorting loss.
  2. Heuristic rewards do not match the first variation of SW22\mathcal{SW}_2^2. Using either an atom-replacement finite difference or a frozen squared residual rSR(x)=−1K∑m(⟨um,x⟩−Tm(⟨um,x⟩))2r_{\mathrm{SR}}(x) = -\frac{1}{K}\sum_m (\langle u_m, x\rangle - T_m(\langle u_m, x\rangle))^2 does not give the exact policy gradient of −SW22(ρπ,ρE)-\mathcal{SW}_2^2(\rho_\pi, \rho_E), because the 1D monotone transport map Tm=Fνm−1∘FμmT_m = F_{\nu_m}^{-1} \circ F_{\mu_m} itself depends on ρπ\rho_\pi.

Recently I wanted a decisive answer: if we strip away all the adversarial heuristics, fix the math completely, and move to massively parallel on-policy GPU RL (IsaacLab), can SWIL be salvaged?


1. The Clean Fix: Exact-Dual SWIL on Fresh On-Policy GPU Batches

Fix KK unit directions U={u1,…,uK}⊂Sd−1U = \{u_1, \dots, u_K\} \subset \mathbb{S}^{d-1}, and let μm=(um)#ρπ\mu_m = (u_m)_\# \rho_\pi and νm=(um)#ρE\nu_m = (u_m)_\# \rho_E be the 1D pushforwards of the policy and expert occupancy measures. For 1D quadratic optimal transport, the exact first variation of 12W22(μm,νm)\frac{1}{2} W_2^2(\mu_m, \nu_m) with respect to μm\mu_m is the Kantorovich potential ϕm:R→R\phi_m : \mathbb{R} \to \mathbb{R} satisfying

ϕm′(z)=z−Tm(z),Tm(z)=Fνm−1 ⁣(Fμm(z)).\phi'_m(z) = z - T_m(z), \qquad T_m(z) = F_{\nu_m}^{-1}\!\big(F_{\mu_m}(z)\big).

On an on-policy GPU PPO rollout batch of N≈32,768N \approx 32{,}768 transitions (2,0482{,}048 parallel envs ×\times 1616 steps), this is trivial to compute exactly:

  1. Standardize features using expert statistics (avoiding tight clipping that kills long-range transport gradients across disjoint supports).
  2. Project policy and expert batches to [K,N][K, N] and [K,M][K, M], sort both on GPU (torch.sort), and linearly interpolate sorted expert quantiles to form Tm(zπ,i)T_m(z_{\pi, i}).
  3. Integrate ϕm′(z)=z−Tm(z)\phi'_m(z) = z - T_m(z) via cumulative trapezoid along the sorted grid, shift each slice by its minimum so ϕm≥0\phi_m \ge 0, scatter back to sample order, and average ϕˉ(x)=1K∑m=1Kϕm(⟨um,x⟩)\bar\phi(x) = \frac{1}{K}\sum_{m=1}^K \phi_m(\langle u_m, x\rangle).

On an RTX 5090, this takes 1.7–2.4 ms per PPO iteration for N=32,768N = 32{,}768, d=162d = 162, K=256K = 256—with no discriminator network, no gradient penalty, and no off-policy replay staleness.

The terminating-MDP trap

In earlier off-policy MuJoCo tests (leanrl), exact-dual SWIL solved fixed-horizon HalfCheetah-v5 (10,238 return) while collapsing on all four early-terminating environments (Hopper, Walker2d, Ant, Humanoid). The reason is structural: ϕ\phi is only unique up to an additive constant, and additive reward shifts are not policy-invariant in variable-horizon terminating MDPs (the classic DAC absorbing-state bias). Zero-centered −ϕˉ-\bar\phi assigns negative reward to half of transitions, incentivizing early termination whenever Q(s,a)<0Q(s, a) < 0.

Mapping the shifted potential to a strictly positive bounded reward,

rSWIL(x)=exp⁡ ⁣(−β (ϕˉ(x)−ϕˉmin⁡))∈(0,1],r_{\text{SWIL}}(x) = \exp\!\big(-\beta\,(\bar\phi(x) - \bar\phi_{\min})\big) \in (0, 1],

immediately eliminates early-termination suicide on IsaacLab humanoids—at the cost of turning quadratic 1D slice tail growth ϕm(z)∼12d(z,supp⁡(νm))2\phi_m(z) \sim \frac{1}{2} d(z, \operatorname{supp}(\nu_m))^2 into a saturating Gaussian tail. (I also tested a polynomial-tailed Cauchy variant 1/(1+β(ϕˉ−ϕˉmin⁡))1/(1 + \beta(\bar\phi - \bar\phi_{\min})) with top-α\alpha slice weighting; it reduced foot skating slightly to 0.088 on Walk, but trailed plain exp_shift on SW2, speed, and survival.)


2. Where It Works—and Where Official AMP Still Wins

To avoid fooling myself with a weak baseline (my first custom LSGAN discriminator failed because a 20× GP scaling mismatch and large head init crippled it), I benchmarked a single frozen SWIL-Linear config (K=256K=256, β=1.0\beta=1.0, exp_shift, zero per-task tuning) directly against IsaacLab’s unmodified official skrl AMP trainer across all three built-in Humanoid-AMP tasks (Walk, Run, Dance; 2,0482{,}048 envs, 1,2001{,}200 iterations, evaluated deterministically over 256256 envs for 300300 steps):

Task (ref speed)MethodEp Len ↑Tail p10 ↑fall < 150 ↓Speed (m/s)Held-out SW2 ↓Foot Skate ↓
Walk (1.52 m/s)Official skrl AMP291.0299.03.1%1.171.3530.103
Walk (1.52 m/s)SWIL-Linear (frozen)268.743.511.3%1.260.8760.113
Run (3.41 m/s)Official skrl AMP240.028.021.9%2.161.4590.171
Run (3.41 m/s)SWIL-Linear (frozen)211.721.032.4%1.051.3210.104
Dance (0.48 m/s)Official skrl AMP261.943.514.5%0.391.6940.179
Dance (0.48 m/s)SWIL-Linear (frozen)62.637.099.6%0.661.2630.267

On steady periodic Walk, SWIL-Linear produces clean, upright bipedal locomotion out of the box—matching official skrl AMP on foot skating (0.113 vs 0.103), tracking reference speed closer (1.26 vs 1.17 m/s), and cutting held-out SW2 by 35% (0.876 vs 1.353) with zero discriminator. It also cleanly beat non-adversarial RBF-MMD and random-Fourier-feature (swil_rff) baselines, both of which collapsed to flat +0.5 alive bonuses (rew_std = 0.034).

Side-by-side 3D skeleton comparison of Reference Walk, SWIL-Linear, and Official skrl AMP

Side-by-side 240-step PhysX rollout comparison on Humanoid-Walk (green: reference mocap; blue: SWIL-Linear; orange: official skrl AMP). Download the animated clip: walk_compare_skeleton.mp4.

Yet the 3-task table exposes the exact ceiling of fixed-slice OT:

  • SWIL-Linear achieved lower held-out SW2 than official skrl AMP on every single task (Walk, Run, and Dance). The optimizer did its job.
  • Even so, official skrl AMP beat SWIL-Linear on closed-loop survival and behavioral fidelity on every task, and the gap widened monotonically with motion complexity: small on periodic Walk, noticeable on high-speed Run (1.05 vs 2.16 m/s), and catastrophic on non-periodic agile Dance (ep_len = 62.6, 99.6% falling before step 150).

3. The Sharpest Falsification: Precision Contact Manipulation

To test the fixed-horizon pure-IL regime (removing termination bias completely), I trained an oracle PPO policy on Isaac-Lift-Cube-Franka-v0 (97.3% train success, 90.0% deterministic demo success across 50 saved trajectories) and ran pure IL with zero task reward:

Method (Isaac-Lift-Cube-Franka-v0, 50 demos)Final Lift Success ↑Peak SuccessTask Return ↑Held-out SW2 ↓Rew ms/iter
Oracle PPO (shaped task reward)0.973 (demo 0.900)0.980107.18—0.0
AMP / GAIL (LSGAN+GP)0.0940.1912.610.45017.4
RBF-MMD0.0040.0080.940.32717.8
SWIL-Linear0.0000.0081.040.3623.5
SWIL-RFF0.0000.0040.880.3783.5
PWIL0.0000.0000.700.5920.6

Both SWIL-Linear and RBF-MMD drove marginal SW2 down to 0.33–0.36 while achieving 0% lift success. In the 44-dimensional (obs, act) vector, grasping and lifting depend on a tiny nonlinear contact subspace (3 cube coordinates coupled with gripper closure). Isotropic random slices weight all 44 coordinates equally, so the policy matches ~40 arm-kinematics coordinates and ignores the cube. A learned discriminator—despite higher marginal SW2 (0.450)—focuses capacity directly on the discriminating contact subspace.


4. Does Exact-Dual Sliced OT Transfer to RLVR Post-Training?

Before closing the book on the exact-dual primitive, I ran one more falsification test in a completely different regime: diversity regularization in RL with verifiable rewards (RLVR). The hypothesis was that a correctness-gated exact-dual SW22\mathcal{SW}_2^2 bonus in a frozen embedding space (matching correct policy rollouts against a reference base-model distribution) could preserve solution diversity and improve held-out pass@16 over plain GRPO+KL without collapsing pass@1.

I tested three matched-budget arms on Qwen/Qwen2.5-1.5B-Instruct trained with LoRA GRPO on MATH (Levels 3–5, 300 steps, kl_beta = 0.005, lr = 5e-5, lambda_div = 0.25) and evaluated on n = 256 held-out MATH-500 prompts (k = 16 samples/prompt, temperature 0.8):

Arm (300 steps, MATH-500, n = 256)pass@1 ↑pass@4 ↑pass@8 ↑pass@16 ↑Correct Emb. eff_rankTotal Correct (/4096)
GRPO + KL0.36600.51990.58520.648443.651,499
GRPO + cosine repulsion0.34860.49120.55050.609443.201,428
GRPO + SW exact-dual0.34060.49450.56070.617243.361,395

Paired bootstrap 95% confidence intervals (10,000 resamples over the 256 matched prompts) confirm a clean negative result:

  • sw_exact_dual − kl: −2.54 pp on pass@1 ([-3.98, -1.12] pp), −2.54 pp on pass@4 ([-4.62, -0.42] pp), and −3.12 pp on pass@16 ([-7.81, +1.17] pp).
  • cosine − kl: −1.73 pp on pass@1 ([-3.15, -0.32] pp) and −3.91 pp on pass@16 ([-7.81, 0.00] pp).
  • sw_exact_dual − cosine: statistically indistinguishable across all kk.

Both embedding-space diversity bonuses distorted within-group GRPO advantages—rewarding surface embedding spread among correct rollouts rather than reliable reasoning—while adding ~25–28% wall-clock overhead and producing no gain in effective rank (43.36 vs 43.65).


5. Closing the Chapter

Together, the robotics and RLVR experiments give a clean, falsifiable verdict on SWIL and close this thread from my thesis:

  • What works: Exact-dual 1D Kantorovich potentials on fresh on-policy GPU batches are genuinely fast (~1.7 ms/iter), stable, and easy to deploy—capable of learning upright periodic bipedal locomotion without an adversarial discriminator.
  • Why it is not a general AMP replacement: Minimizing marginal linear SW22\mathcal{SW}_2^2 does not guarantee closed-loop physical competence when success depends on nonlinear dynamical couplings (Dance, high-speed Run), sparse contact subspaces (Franka Lift), or support containment rather than mass conservation.
  • Why it did not transfer to RLVR: In verifier-driven post-training, adding an embedding-space sliced-OT bonus on top of binary correctness rewards degraded pass@1 through pass@16 relative to plain GRPO+KL.
  • Where the primitive still belongs: Only when global distribution matching in a compact, dense, task-aligned feature space is literally the objective (e.g., matching unlabeled telemetry distributions for sim-to-real calibration)—not as a drop-in proxy for contact physics or reasoning diversity.
Reinforcement LearningImitation LearningOptimal TransportRoboticsPost-Training