Tsukada Lab. Computer Network Group
← 発表論文一覧 ワークショップ等

Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition

Yun Li, Ehsan Javanmardi, Yidu Zhang, Simon Thompson, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada

Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, 2026年7月6日

3行要約

  • 安全性、効率、快適性、実行時間に関する環境からのフィードバックで、拡散プランナを微調整する。
  • 二値の安全報酬が持つ学習上の制約を示し、滑らかな代理報酬と厳密なマージン罰則の組み合わせを検証する。
  • 地域に共通する幾何情報と、運行設計領域ごとのノイズ除去の挙動を分けて適応させる。

アブストラクト(原文)

We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.

引用方法

Y. Li, E. Javanmardi, Y. Zhang, S. Thompson, Q. Zhang, Z. Zeng, S. Liu, P. Wang, Z. Guo, and M. Tsukada, “Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition,” in Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, 2026.

BibTeX

@workshop{Li2026d,
  title         = {Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition},
  author        = {Yun Li and Ehsan Javanmardi and Yidu Zhang and Simon Thompson and Qunli Zhang and Zifan Zeng and Shiming Liu and Peng Wang and Zixuan Guo and Manabu Tsukada},
  url           = {https://sites.google.com/view/rlxf-icml2026},
  year          = {2026},
  date          = {2026-07-06},
  urldate       = {2026-07-06},
  booktitle     = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
  address       = {Seoul, Korea},
  abstract      = {We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.},
  howpublished  = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
  keywords      = {ITS},
  pubstate      = {published},
  tppubtype     = {workshop}
}