Tsukada Lab. Computer Network Group

Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning

Yun Li

Ph.D. Thesis, The University of Tokyo, Sep 30, 2026

Summary in three lines

  • This thesis studies safer behavior and real-time inference in generative driving models.
  • It develops preference alignment and causal scene descriptions for vision-language-action driving.
  • For diffusion planning, it evaluates modular inference, deadline handling and adaptation across regions in closed-loop tests.

Figures from the paper

Figure 1.1: Paradigms of autonomous-driving decision making
Figure 1.1. Paradigms of autonomous-driving decision makingSource: Figure 1.1 from Ph.D. thesis, The University of Tokyo, 2026 (author's version)Rule- and optimization-based methods are interpretable but do not generalize to the long tail, and learning-based methods inherit a long-tail and interpretability gap. The thesis targets the knowledge-based paradigm, in which generative models reason over scenes and language instructions.
Figure 3.3: PrefDrive system overview
Figure 3.3. PrefDrive system overviewSource: Figure 3.3 from Ph.D. thesis, The University of Tokyo, 2026 (author's version)A vision-language-action model takes camera and LiDAR input with a navigation instruction and outputs driving actions in CARLA. Only the LoRA adapters are fine-tuned with direct preference optimization on chosen and rejected driving responses.
Figure 5.1: Modular Diffusion Planner architecture
Figure 5.1. Modular Diffusion Planner architectureSource: Figure 5.1 from Ph.D. thesis, The University of Tokyo, 2026 (author's version)The 18,398-node monolithic planner graph is split into an encoder run once, a denoiser repeated N times by a native C++ DPM-Solver++, and a turn-indicator head. The result is a drop-in replacement for the Autoware planner.

Abstract

Vision-Language-Action (VLA) models and diffusion-based motion planners are now competitive learning paradigms for autonomous driving. They interpret natural-language instructions, reason about long-tail scenarios, and produce diverse multimodal trajectories that classical modular pipelines cannot easily express. Despite strong results on academic benchmarks, two gaps prevent direct deployment on production self-driving stacks. First, a VLA trained with supervised fine-tuning imitates the average behavior of the demonstration corpus and has no notion of risk; in safety-critical situations it fails silently. Second, monolithic diffusion planners ship as a single ONNX graph with a fixed number of denoising steps and provide no real-time guarantee on commodity automotive compute. This thesis addresses both gaps and demonstrates that the resulting models can be integrated into the Autoware production stack and the CARLA closed-loop benchmark. The first research line develops a preference-alignment framework for VLA driving policies. Starting from PrefDrive, we apply Direct Preference Optimization (DPO) to a LLaMA-based driving policy with a 74,040-sequence preference dataset, reducing traffic-light violations by 28.1% and layout collisions by 63.5% on CARLA. Because pairwise preferences cannot capture the multi-way risk ranking that a human safety driver naturally expresses, we extend the framework to Multi-PrefDrive, which fits a Plackett–Luce ranking model on 148,080 ranked trajectory lists and obtains an 11.0% improvement in driving score and an 83.6% reduction in infrastructure collisions. To prevent probability collapse and to weight rare safety-critical scenes appropriately, PL-DPO-NLL combines a scene-adaptive β coefficient with an NLL regularizer that keeps the reference distribution non-degenerate during ranking optimization. On the CARLA closed-loop benchmark, PL-DPO-NLL attains a Driving Score of 58.26 (+8.4% over the strongest baseline), a Route Completion of 65.9%, and the highest Infraction Penalty (0.891) among all preference methods evaluated. Causal Scene Narration (CSN) rewrites the VLA’s language input under three principles, intent–constraint causal alignment, quantitative physical grounding, and structured information separation, and pairs it with a runtime safety supervisor that monitors directional collisions, lateral deviation, and time-to-collision, with fallback to a classical planner. On an eight-town closed-loop CARLA aggregate, CSN improves Driving Score by 31.1% on the original LMDrive backbone (from 32.54 to 42.67) and by 24.5% on the preference-aligned variant, with Infraction Score rising from 0.729 to 0.787, and requires no retraining and no additional GPU memory. The second research line targets four orthogonal constraints that prevent diffusion-based motion planners from running in a production stack. The Modular Diffusion Benchmark, accepted to ITSC 2026, decomposes the 18,398-node monolithic ONNX graph into three subgraphs (encoder, denoiser, post-processing) via NVIDIA GraphSurgeon, re-implements DPM-Solver++ in C++ with first- and second-order solvers and variance-preserving continuous noise schedules, and integrates as a drop-in ROS 2 Humble node compatible with Autoware and the AWSIM simulator; encoder-cache reuse alone reduces per-cycle latency at three denoising steps from 119 ms to 37 ms (3.2×) at equivalent CPU cost. On top of this modular substrate, Anytime Diffusion Planning introduces budget-adaptive denoising: a Budget-Aware Step Selector chooses the largest feasible step count under a per-cycle deadline, a Convergence Monitor early-exits when the trajectory has stabilized, and a Deadline Guard ensures that a valid plan is emitted on every control cycle. The deadline-miss rate drops from 74.5% to 0.1% with zero collisions in closed-loop simulation under injected upstream jitter. SE-LoRA shows that reward fine-tuning of diffusion planners decomposes into a cross-ODD aligned shared subspace plus per-ODD expert subspaces, with a 10× gradient-cosine gap and a Singapore↔Pittsburgh causal-patching double dissociation; the 1:2 shared-to-expert rank ratio is predicted by the cross-ODD PCA spectral gap, and a differentiable mirror of nuPlan closed-loop scoring (PCDR) whose forward pass is bit-exact on the collision, progress, and driving-direction sub-scores is introduced together with a formalized forward-fidelity vs. gradient-reach no-go result. GeoFlex proves that cross-region routing across per-region experts has an architectural ceiling (no scene-feature router beats a static-prior baseline), and shows that joint replay outside the routing subspace breaks it, reaching 4.26 m against 13.74 m fleet-mean maximum lateral offset. The integrated stack is released as an open-source benchmark. Two survey papers situate this work within the broader research landscape. The first organizes more than 100 LLM-based driving methods into a five-pillar taxonomy spanning perception, prediction, planning, control, and end-to-end frameworks. The second provides a critical analysis of scaling laws (which invert relative to Chinchilla in NLP), GRPO-style alignment, action tokenization strategies, and the safety and regulatory landscape (ISO/PAS 8800, EU/US/China regulations) for driving foundation models. The contributions are threefold. First, a unified framework that advances preference alignment for VLA driving from binary DPO through Plackett–Luce ranking to scene-adaptive PL-DPO-NLL, integrated with a causal-narration and runtime-safety layer that enforces safe behavior at both training and inference time. Second, a four-pillar diffusion-planning substrate (modular ONNX, anytime scheduling, SE-LoRA reward fine-tuning with PCDR, and GeoFlex cross-region adaptation) that makes diffusion-based motion planners real-time compatible, reward-alignable, and cross-region deployable on production autonomous-driving stacks. Third, two surveys that organize the research landscape and identify the open challenges this thesis addresses. The two research lines share a common philosophy: take a generative model that excels on a benchmark, decompose it into components whose safety and timing properties can be analyzed and controlled, and re-compose it so that the resulting policy is safe enough, fast enough, and verifiable enough to be deployed. This research is the foundational work for the author’s JSPS Research Fellowship for Young Scientists (DC2) project (2026–2028), titled Large Language Model Reinforcement Learning Combining Fast and Slow Thinking for Driving Decisions. The fast–slow architecture envisioned in DC2 builds directly on the preference-alignment and modular-diffusion contributions of this thesis, with closed-loop validation extended from CARLA simulation to the laboratory’s PIXKIT 3.0 vehicle at the University of Tokyo Kashiwa campus.

How to cite

Y. Li, “Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning,” in Ph.D. Thesis, The University of Tokyo, 2026.

BibTeX

@phdthesis{Li2026f,
  title         = {Preference-Aligned Generative Models for Safe and Efficient Autonomous Driving: Vision-Language-Action Models and Real-Time Diffusion Planning},
  author        = {Yun Li},
  year          = {2026},
  date          = {2026-09-30},
  urldate       = {2026-09-30},
  school        = {Ph.D. Thesis, The University of Tokyo},
  abstract      = {Vision-Language-Action (VLA) models and diffusion-based motion planners are now competitive learning paradigms for autonomous driving. They interpret natural-language instructions, reason about long-tail scenarios, and produce diverse multimodal trajectories that classical modular pipelines cannot easily express. Despite strong results on academic benchmarks, two gaps prevent direct deployment on production self-driving stacks. First, a VLA trained with supervised fine-tuning imitates the average behavior of the demonstration corpus and has no notion of risk; in safety-critical situations it fails silently. Second, monolithic diffusion planners ship as a single ONNX graph with a fixed number of denoising steps and provide no real-time guarantee on commodity automotive compute. This thesis addresses both gaps and demonstrates that the resulting models can be integrated into the Autoware production stack and the CARLA closed-loop benchmark. The first research line develops a preference-alignment framework for VLA driving policies. Starting from PrefDrive, we apply Direct Preference Optimization (DPO) to a LLaMA-based driving policy with a 74,040-sequence preference dataset, reducing traffic-light violations by 28.1% and layout collisions by 63.5% on CARLA. Because pairwise preferences cannot capture the multi-way risk ranking that a human safety driver naturally expresses, we extend the framework to Multi-PrefDrive, which fits a Plackett–Luce ranking model on 148,080 ranked trajectory lists and obtains an 11.0% improvement in driving score and an 83.6% reduction in infrastructure collisions. To prevent probability collapse and to weight rare safety-critical scenes appropriately, PL-DPO-NLL combines a scene-adaptive β coefficient with an NLL regularizer that keeps the reference distribution non-degenerate during ranking optimization. On the CARLA closed-loop benchmark, PL-DPO-NLL attains a Driving Score of 58.26 (+8.4% over the strongest baseline), a Route Completion of 65.9%, and the highest Infraction Penalty (0.891) among all preference methods evaluated. Causal Scene Narration (CSN) rewrites the VLA’s language input under three principles, intent–constraint causal alignment, quantitative physical grounding, and structured information separation, and pairs it with a runtime safety supervisor that monitors directional collisions, lateral deviation, and time-to-collision, with fallback to a classical planner. On an eight-town closed-loop CARLA aggregate, CSN improves Driving Score by 31.1% on the original LMDrive backbone (from 32.54 to 42.67) and by 24.5% on the preference-aligned variant, with Infraction Score rising from 0.729 to 0.787, and requires no retraining and no additional GPU memory. The second research line targets four orthogonal constraints that prevent diffusion-based motion planners from running in a production stack. The Modular Diffusion Benchmark, accepted to ITSC 2026, decomposes the 18,398-node monolithic ONNX graph into three subgraphs (encoder, denoiser, post-processing) via NVIDIA GraphSurgeon, re-implements DPM-Solver++ in C++ with first- and second-order solvers and variance-preserving continuous noise schedules, and integrates as a drop-in ROS 2 Humble node compatible with Autoware and the AWSIM simulator; encoder-cache reuse alone reduces per-cycle latency at three denoising steps from 119 ms to 37 ms (3.2×) at equivalent CPU cost. On top of this modular substrate, Anytime Diffusion Planning introduces budget-adaptive denoising: a Budget-Aware Step Selector chooses the largest feasible step count under a per-cycle deadline, a Convergence Monitor early-exits when the trajectory has stabilized, and a Deadline Guard ensures that a valid plan is emitted on every control cycle. The deadline-miss rate drops from 74.5% to 0.1% with zero collisions in closed-loop simulation under injected upstream jitter. SE-LoRA shows that reward fine-tuning of diffusion planners decomposes into a cross-ODD aligned shared subspace plus per-ODD expert subspaces, with a 10× gradient-cosine gap and a Singapore↔Pittsburgh causal-patching double dissociation; the 1:2 shared-to-expert rank ratio is predicted by the cross-ODD PCA spectral gap, and a differentiable mirror of nuPlan closed-loop scoring (PCDR) whose forward pass is bit-exact on the collision, progress, and driving-direction sub-scores is introduced together with a formalized forward-fidelity vs. gradient-reach no-go result. GeoFlex proves that cross-region routing across per-region experts has an architectural ceiling (no scene-feature router beats a static-prior baseline), and shows that joint replay outside the routing subspace breaks it, reaching 4.26 m against 13.74 m fleet-mean maximum lateral offset. The integrated stack is released as an open-source benchmark. Two survey papers situate this work within the broader research landscape. The first organizes more than 100 LLM-based driving methods into a five-pillar taxonomy spanning perception, prediction, planning, control, and end-to-end frameworks. The second provides a critical analysis of scaling laws (which invert relative to Chinchilla in NLP), GRPO-style alignment, action tokenization strategies, and the safety and regulatory landscape (ISO/PAS 8800, EU/US/China regulations) for driving foundation models. The contributions are threefold. First, a unified framework that advances preference alignment for VLA driving from binary DPO through Plackett–Luce ranking to scene-adaptive PL-DPO-NLL, integrated with a causal-narration and runtime-safety layer that enforces safe behavior at both training and inference time. Second, a four-pillar diffusion-planning substrate (modular ONNX, anytime scheduling, SE-LoRA reward fine-tuning with PCDR, and GeoFlex cross-region adaptation) that makes diffusion-based motion planners real-time compatible, reward-alignable, and cross-region deployable on production autonomous-driving stacks. Third, two surveys that organize the research landscape and identify the open challenges this thesis addresses. The two research lines share a common philosophy: take a generative model that excels on a benchmark, decompose it into components whose safety and timing properties can be analyzed and controlled, and re-compose it so that the resulting policy is safe enough, fast enough, and verifiable enough to be deployed. This research is the foundational work for the author’s JSPS Research Fellowship for Young Scientists (DC2) project (2026–2028), titled Large Language Model Reinforcement Learning Combining Fast and Slow Thinking for Driving Decisions. The fast–slow architecture envisioned in DC2 builds directly on the preference-alignment and modular-diffusion contributions of this thesis, with closed-loop validation extended from CARLA simulation to the laboratory’s PIXKIT 3.0 vehicle at the University of Tokyo Kashiwa campus.},
  keywords      = {Thesis},
  pubstate      = {published},
  tppubtype     = {phdthesis}
}