Tsukada Lab. Computer Network Group
← Publications Workshop / WiP

Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition

Yun Li, Ehsan Javanmardi, Yidu Zhang, Simon Thompson, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada

Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, Jul 6, 2026

Summary in three lines

  • The study fine-tunes diffusion planners using feedback about safety, efficiency, comfort and runtime.
  • It identifies a limitation of hard binary safety rewards and tests a smooth surrogate with an exact-margin penalty.
  • The proposed adaptation separates shared geometric knowledge from denoising behavior specific to each operating domain.

Abstract

We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.

How to cite

Y. Li, E. Javanmardi, Y. Zhang, S. Thompson, Q. Zhang, Z. Zeng, S. Liu, P. Wang, Z. Guo, and M. Tsukada, “Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition,” in Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, 2026.

BibTeX

@workshop{Li2026d,
  title         = {Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition},
  author        = {Yun Li and Ehsan Javanmardi and Yidu Zhang and Simon Thompson and Qunli Zhang and Zifan Zeng and Shiming Liu and Peng Wang and Zixuan Guo and Manabu Tsukada},
  url           = {https://sites.google.com/view/rlxf-icml2026},
  year          = {2026},
  date          = {2026-07-06},
  urldate       = {2026-07-06},
  booktitle     = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
  address       = {Seoul, Korea},
  abstract      = {We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.},
  howpublished  = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
  keywords      = {ITS},
  pubstate      = {published},
  tppubtype     = {workshop}
}