Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition
Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, 2026年7月6日
3行要約
- 安全性、効率、快適性、実行時間に関する環境からのフィードバックで、拡散プランナを微調整する。
- 二値の安全報酬が持つ学習上の制約を示し、滑らかな代理報酬と厳密なマージン罰則の組み合わせを検証する。
- 地域に共通する幾何情報と、運行設計領域ごとのノイズ除去の挙動を分けて適応させる。
アブストラクト(原文)
We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.
引用方法
Y. Li, E. Javanmardi, Y. Zhang, S. Thompson, Q. Zhang, Z. Zeng, S. Liu, P. Wang, Z. Guo, and M. Tsukada, “Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition,” in Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026), Seoul, Korea, 2026.
BibTeX
@workshop{Li2026d,
title = {Learning Diffusion Planners from World Feedback: A No-Go Result on Bit-Exact Safety Rewards and an ODD-Adaptive Shared/Expert Decomposition},
author = {Yun Li and Ehsan Javanmardi and Yidu Zhang and Simon Thompson and Qunli Zhang and Zifan Zeng and Shiming Liu and Peng Wang and Zixuan Guo and Manabu Tsukada},
url = {https://sites.google.com/view/rlxf-icml2026},
year = {2026},
date = {2026-07-06},
urldate = {2026-07-06},
booktitle = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
address = {Seoul, Korea},
abstract = {We fine-tune diffusion-based trajectory planners using closed-loop safety, efficiency, comfort, and real-time deployment outcomes as reward, learning from world feedback rather than human preferences. Across 8 operational design domains (ODDs) in two independent stacks (Autoware + nuPlan, different sensors and planner backbones), world-grounded fine-tuning yields closed-loop gains within seed-to-seed noise but reproducibly exposes a Shared-Expert LoRA (SE-LoRA) structure: the reward gradient splits into a cross-ODD aligned subspace (geometric priors) and ODD-specific subspaces (denoising dynamics). Three independent signatures support the split: a Singapore $leftrightarrow$ Pittsburgh double dissociation under causal activation patching ($rho = -0.96$), a $10times$ gradient-cosine gap between components, and a cross-ODD PCA spectral gap that predicts a $1:2$ shared-to-expert rank ratio, confirmed by rank-allocation sweeps on both stacks. The same lens also produces a no-go result relevant to any world-grounded reward with hard safety thresholds: PCDR (Per-Closed-Loop Differentiable Reward), to our knowledge the first bit-exact differentiable mirror of nuPlan closed-loop scoring, exposes a forward-fidelity vs. gradient-reach trade-off for $0/1$ safety rewards, motivating a constrained-recovery design (smooth CLS proxy + bit-exact PCDR-derived margin penalty). We release PCDR, SE-LoRA, and anonymized real-vehicle deployment data (uncommon in the diffusion-planning literature) covering both simulation and a production ROS2 stack. World-grounded rewards reveal structural decompositions that BC-only training hides and impose hard limits on $0/1$ safety signals; the spectral-gap recipe predicts LoRA ranks without sweep, and PCDR provides a standalone audit of reward $leftrightarrow$ metric alignment.},
howpublished = {Workshop on Reinforcement Learning from World Feedback (RLxF) at Forty-Third International Conference on Machine Learning (ICML 2026)},
keywords = {ITS},
pubstate = {published},
tppubtype = {workshop}
}