Tsukada Lab. Computer Network Group

Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning

Yun Li

Ph.D. Thesis, The University of Tokyo, 2026年9月30日

3行要約

  • 生成型自動運転モデルの安全性とリアルタイム推論を扱う博士論文である。
  • 視覚言語行動モデルに対する選好整合と、因果関係を明記した場面記述を開発した。
  • 拡散プランナでは、推論のモジュール化、期限への対応、地域間適応を閉ループで評価した。

論文の図

図 1.1: 自動運転の意思決定手法の変遷
図 1.1. 自動運転の意思決定手法の変遷出典: 博士論文、東京大学、2026(著者版) の図 1.1ルールや最適化に基づく手法は解釈しやすいが稀な状況に汎化しにくく、学習に基づく手法にも同じ課題と解釈性の課題が残る。本論文は、生成モデルがシーンと言語指示から推論する知識ベースの手法を対象とする。
図 3.3: PrefDriveのシステム構成
図 3.3. PrefDriveのシステム構成出典: 博士論文、東京大学、2026(著者版) の図 3.3Vision-Language-Actionモデルがカメラ・LiDARの入力とナビゲーション指示から、CARLA上での運転操作を出力する。選ばれた応答と退けられた応答の組で直接選好最適化(DPO)を行い、LoRAアダプタだけを更新する。
図 5.1: モジュール化した拡散プランナーの構成
図 5.1. モジュール化した拡散プランナーの構成出典: 博士論文、東京大学、2026(著者版) の図 5.118,398ノードの単一の計算グラフを、1回だけ実行するエンコーダ、C++で実装したDPM-Solver++でN回繰り返すデノイザ、方向指示器の出力部に分割した。Autowareのプランナーとそのまま置き換えて使える。

アブストラクト(原文)

Vision-Language-Action (VLA) models and diffusion-based motion planners are now competitive learning paradigms for autonomous driving. They interpret natural-language instructions, reason about long-tail scenarios, and produce diverse multimodal trajectories that classical modular pipelines cannot easily express. Despite strong results on academic benchmarks, two gaps prevent direct deployment on production self-driving stacks. First, a VLA trained with supervised fine-tuning imitates the average behavior of the demonstration corpus and has no notion of risk; in safety-critical situations it fails silently. Second, monolithic diffusion planners ship as a single ONNX graph with a fixed number of denoising steps and provide no real-time guarantee on commodity automotive compute. This thesis addresses both gaps and demonstrates that the resulting models can be integrated into the Autoware production stack and the CARLA closed-loop benchmark. The first research line develops a preference-alignment framework for VLA driving policies. Starting from PrefDrive, we apply Direct Preference Optimization (DPO) to a LLaMA-based driving policy with a 74,040-sequence preference dataset, reducing traffic-light violations by 28.1% and layout collisions by 63.5% on CARLA. Because pairwise preferences cannot capture the multi-way risk ranking that a human safety driver naturally expresses, we extend the framework to Multi-PrefDrive, which fits a Plackett–Luce ranking model on 148,080 ranked trajectory lists and obtains an 11.0% improvement in driving score and an 83.6% reduction in infrastructure collisions. To prevent probability collapse and to weight rare safety-critical scenes appropriately, PL-DPO-NLL combines a scene-adaptive β coefficient with an NLL regularizer that keeps the reference distribution non-degenerate during ranking optimization. On the CARLA closed-loop benchmark, PL-DPO-NLL attains a Driving Score of 58.26 (+8.4% over the strongest baseline), a Route Completion of 65.9%, and the highest Infraction Penalty (0.891) among all preference methods evaluated. Causal Scene Narration (CSN) rewrites the VLA’s language input under three principles, intent–constraint causal alignment, quantitative physical grounding, and structured information separation, and pairs it with a runtime safety supervisor that monitors directional collisions, lateral deviation, and time-to-collision, with fallback to a classical planner. On an eight-town closed-loop CARLA aggregate, CSN improves Driving Score by 31.1% on the original LMDrive backbone (from 32.54 to 42.67) and by 24.5% on the preference-aligned variant, with Infraction Score rising from 0.729 to 0.787, and requires no retraining and no additional GPU memory. The second research line targets four orthogonal constraints that prevent diffusion-based motion planners from running in a production stack. The Modular Diffusion Benchmark, accepted to ITSC 2026, decomposes the 18,398-node monolithic ONNX graph into three subgraphs (encoder, denoiser, post-processing) via NVIDIA GraphSurgeon, re-implements DPM-Solver++ in C++ with first- and second-order solvers and variance-preserving continuous noise schedules, and integrates as a drop-in ROS 2 Humble node compatible with Autoware and the AWSIM simulator; encoder-cache reuse alone reduces per-cycle latency at three denoising steps from 119 ms to 37 ms (3.2×) at equivalent CPU cost. On top of this modular substrate, Anytime Diffusion Planning introduces budget-adaptive denoising: a Budget-Aware Step Selector chooses the largest feasible step count under a per-cycle deadline, a Convergence Monitor early-exits when the trajectory has stabilized, and a Deadline Guard ensures that a valid plan is emitted on every control cycle. The deadline-miss rate drops from 74.5% to 0.1% with zero collisions in closed-loop simulation under injected upstream jitter. SE-LoRA shows that reward fine-tuning of diffusion planners decomposes into a cross-ODD aligned shared subspace plus per-ODD expert subspaces, with a 10× gradient-cosine gap and a Singapore↔Pittsburgh causal-patching double dissociation; the 1:2 shared-to-expert rank ratio is predicted by the cross-ODD PCA spectral gap, and a differentiable mirror of nuPlan closed-loop scoring (PCDR) whose forward pass is bit-exact on the collision, progress, and driving-direction sub-scores is introduced together with a formalized forward-fidelity vs. gradient-reach no-go result. GeoFlex proves that cross-region routing across per-region experts has an architectural ceiling (no scene-feature router beats a static-prior baseline), and shows that joint replay outside the routing subspace breaks it, reaching 4.26 m against 13.74 m fleet-mean maximum lateral offset. The integrated stack is released as an open-source benchmark. Two survey papers situate this work within the broader research landscape. The first organizes more than 100 LLM-based driving methods into a five-pillar taxonomy spanning perception, prediction, planning, control, and end-to-end frameworks. The second provides a critical analysis of scaling laws (which invert relative to Chinchilla in NLP), GRPO-style alignment, action tokenization strategies, and the safety and regulatory landscape (ISO/PAS 8800, EU/US/China regulations) for driving foundation models. The contributions are threefold. First, a unified framework that advances preference alignment for VLA driving from binary DPO through Plackett–Luce ranking to scene-adaptive PL-DPO-NLL, integrated with a causal-narration and runtime-safety layer that enforces safe behavior at both training and inference time. Second, a four-pillar diffusion-planning substrate (modular ONNX, anytime scheduling, SE-LoRA reward fine-tuning with PCDR, and GeoFlex cross-region adaptation) that makes diffusion-based motion planners real-time compatible, reward-alignable, and cross-region deployable on production autonomous-driving stacks. Third, two surveys that organize the research landscape and identify the open challenges this thesis addresses. The two research lines share a common philosophy: take a generative model that excels on a benchmark, decompose it into components whose safety and timing properties can be analyzed and controlled, and re-compose it so that the resulting policy is safe enough, fast enough, and verifiable enough to be deployed. This research is the foundational work for the author’s JSPS Research Fellowship for Young Scientists (DC2) project (2026–2028), titled Large Language Model Reinforcement Learning Combining Fast and Slow Thinking for Driving Decisions. The fast–slow architecture envisioned in DC2 builds directly on the preference-alignment and modular-diffusion contributions of this thesis, with closed-loop validation extended from CARLA simulation to the laboratory’s PIXKIT 3.0 vehicle at the University of Tokyo Kashiwa campus.

引用方法

Y. Li, “Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning,” in Ph.D. Thesis, The University of Tokyo, 2026.

BibTeX

@phdthesis{Li2026f,
  title         = {Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning},
  author        = {Yun Li},
  year          = {2026},
  date          = {2026-09-30},
  urldate       = {2026-09-30},
  school        = {Ph.D. Thesis, The University of Tokyo},
  abstract      = {Vision-Language-Action (VLA) models and diffusion-based motion planners are now competitive learning paradigms for autonomous driving. They interpret natural-language instructions, reason about long-tail scenarios, and produce diverse multimodal trajectories that classical modular pipelines cannot easily express. Despite strong results on academic benchmarks, two gaps prevent direct deployment on production self-driving stacks. First, a VLA trained with supervised fine-tuning imitates the average behavior of the demonstration corpus and has no notion of risk; in safety-critical situations it fails silently. Second, monolithic diffusion planners ship as a single ONNX graph with a fixed number of denoising steps and provide no real-time guarantee on commodity automotive compute. This thesis addresses both gaps and demonstrates that the resulting models can be integrated into the Autoware production stack and the CARLA closed-loop benchmark. The first research line develops a preference-alignment framework for VLA driving policies. Starting from PrefDrive, we apply Direct Preference Optimization (DPO) to a LLaMA-based driving policy with a 74,040-sequence preference dataset, reducing traffic-light violations by 28.1% and layout collisions by 63.5% on CARLA. Because pairwise preferences cannot capture the multi-way risk ranking that a human safety driver naturally expresses, we extend the framework to Multi-PrefDrive, which fits a Plackett–Luce ranking model on 148,080 ranked trajectory lists and obtains an 11.0% improvement in driving score and an 83.6% reduction in infrastructure collisions. To prevent probability collapse and to weight rare safety-critical scenes appropriately, PL-DPO-NLL combines a scene-adaptive β coefficient with an NLL regularizer that keeps the reference distribution non-degenerate during ranking optimization. On the CARLA closed-loop benchmark, PL-DPO-NLL attains a Driving Score of 58.26 (+8.4% over the strongest baseline), a Route Completion of 65.9%, and the highest Infraction Penalty (0.891) among all preference methods evaluated. Causal Scene Narration (CSN) rewrites the VLA’s language input under three principles, intent–constraint causal alignment, quantitative physical grounding, and structured information separation, and pairs it with a runtime safety supervisor that monitors directional collisions, lateral deviation, and time-to-collision, with fallback to a classical planner. On an eight-town closed-loop CARLA aggregate, CSN improves Driving Score by 31.1% on the original LMDrive backbone (from 32.54 to 42.67) and by 24.5% on the preference-aligned variant, with Infraction Score rising from 0.729 to 0.787, and requires no retraining and no additional GPU memory. The second research line targets four orthogonal constraints that prevent diffusion-based motion planners from running in a production stack. The Modular Diffusion Benchmark, accepted to ITSC 2026, decomposes the 18,398-node monolithic ONNX graph into three subgraphs (encoder, denoiser, post-processing) via NVIDIA GraphSurgeon, re-implements DPM-Solver++ in C++ with first- and second-order solvers and variance-preserving continuous noise schedules, and integrates as a drop-in ROS 2 Humble node compatible with Autoware and the AWSIM simulator; encoder-cache reuse alone reduces per-cycle latency at three denoising steps from 119 ms to 37 ms (3.2×) at equivalent CPU cost. On top of this modular substrate, Anytime Diffusion Planning introduces budget-adaptive denoising: a Budget-Aware Step Selector chooses the largest feasible step count under a per-cycle deadline, a Convergence Monitor early-exits when the trajectory has stabilized, and a Deadline Guard ensures that a valid plan is emitted on every control cycle. The deadline-miss rate drops from 74.5% to 0.1% with zero collisions in closed-loop simulation under injected upstream jitter. SE-LoRA shows that reward fine-tuning of diffusion planners decomposes into a cross-ODD aligned shared subspace plus per-ODD expert subspaces, with a 10× gradient-cosine gap and a Singapore↔Pittsburgh causal-patching double dissociation; the 1:2 shared-to-expert rank ratio is predicted by the cross-ODD PCA spectral gap, and a differentiable mirror of nuPlan closed-loop scoring (PCDR) whose forward pass is bit-exact on the collision, progress, and driving-direction sub-scores is introduced together with a formalized forward-fidelity vs. gradient-reach no-go result. GeoFlex proves that cross-region routing across per-region experts has an architectural ceiling (no scene-feature router beats a static-prior baseline), and shows that joint replay outside the routing subspace breaks it, reaching 4.26 m against 13.74 m fleet-mean maximum lateral offset. The integrated stack is released as an open-source benchmark. Two survey papers situate this work within the broader research landscape. The first organizes more than 100 LLM-based driving methods into a five-pillar taxonomy spanning perception, prediction, planning, control, and end-to-end frameworks. The second provides a critical analysis of scaling laws (which invert relative to Chinchilla in NLP), GRPO-style alignment, action tokenization strategies, and the safety and regulatory landscape (ISO/PAS 8800, EU/US/China regulations) for driving foundation models. The contributions are threefold. First, a unified framework that advances preference alignment for VLA driving from binary DPO through Plackett–Luce ranking to scene-adaptive PL-DPO-NLL, integrated with a causal-narration and runtime-safety layer that enforces safe behavior at both training and inference time. Second, a four-pillar diffusion-planning substrate (modular ONNX, anytime scheduling, SE-LoRA reward fine-tuning with PCDR, and GeoFlex cross-region adaptation) that makes diffusion-based motion planners real-time compatible, reward-alignable, and cross-region deployable on production autonomous-driving stacks. Third, two surveys that organize the research landscape and identify the open challenges this thesis addresses. The two research lines share a common philosophy: take a generative model that excels on a benchmark, decompose it into components whose safety and timing properties can be analyzed and controlled, and re-compose it so that the resulting policy is safe enough, fast enough, and verifiable enough to be deployed. This research is the foundational work for the author’s JSPS Research Fellowship for Young Scientists (DC2) project (2026–2028), titled Large Language Model Reinforcement Learning Combining Fast and Slow Thinking for Driving Decisions. The fast–slow architecture envisioned in DC2 builds directly on the preference-alignment and modular-diffusion contributions of this thesis, with closed-loop validation extended from CARLA simulation to the laboratory’s PIXKIT 3.0 vehicle at the University of Tokyo Kashiwa campus.},
  keywords      = {Thesis},
  pubstate      = {published},
  tppubtype     = {phdthesis}
}