Tsukada Lab. Computer Network Group

Multi-PrefDrive: Optimizing Large Language Models for Autonomous Driving Through Multi-Preference Tuning

Yun Li, Ehsan Javanmardi, Simon Thompson, Kai Katsumata, Alex Orsholits, Manabu Tsukada

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), Hangzhou, China, 2025年10月19日

3行要約

  • Multi-PrefDriveは複数の運転候補の順位で言語モデルを微調整する。
  • Plackett–Luceの選好目的を使い、不適切な候補を結果の重大さに応じて区別する。
  • CARLAでは比較手法より運転スコアが向上し、インフラとの衝突が減少した。

論文の図

図 1: Multi-PrefDriveの構成
図 1. Multi-PrefDriveの構成出典: Li et al., IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025) (2025)(著者原稿の図版) の図 1順位付きの運転選好が、モデルの学習と意思決定のどこで使われるかを示す。

アブストラクト(原文)

This paper introduces Multi-PrefDrive, a framework that significantly enhances LLM-based autonomous driving through multidimensional preference tuning. Aligning LLMs with human driving preferences is crucial yet challenging, as driving scenarios involve complex decisions where multiple incorrect actions can correspond to a single correct choice. Traditional binary preference tuning fails to capture this complexity. Our approach pairs each chosen action with multiple rejected alternatives, better reflecting real-world driving decisions. By implementing the Plackett-Luce preference model, we enable nuanced ranking of actions across the spectrum of possible errors. Experiments in the CARLA simulator demonstrate that our algorithm achieves an 11.0% improvement in overall score and an 83.6% reduction in infrastructure collisions, while showing perfect compliance with traffic signals in certain environments. Comparative analysis against DPO and its variants reveals that Multi-PrefDrive’s superior discrimination between chosen and rejected actions, which achieving a margin value of 25, and such ability has been directly translates to enhanced driving performance. We implement memory-efficient techniques including LoRA and 4-bit quantization to enable deployment on consumer-grade hardware and will open-source our training code and multi-rejected dataset to advance research in LLM-based autonomous driving systems. Project Page (https://liyun0607.github.io/)

動画

関連する動画: Multi-PrefDrive:マルチ嗜好学習によるLLMベース自動運転の高度化

関連する研究・ソフトウェア

引用方法

Y. Li, E. Javanmardi, S. Thompson, K. Katsumata, A. Orsholits, and M. Tsukada, “Multi-PrefDrive: Optimizing Large Language Models for Autonomous Driving Through Multi-Preference Tuning,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), Hangzhou, China, 2025, doi: 10.1109/IROS60139.2025.11247608.

BibTeX

@inproceedings{Li2025d,
  title         = {Multi-PrefDrive: Optimizing Large Language Models for Autonomous Driving Through Multi-Preference Tuning},
  author        = {Yun Li and Ehsan Javanmardi and Simon Thompson and Kai Katsumata and Alex Orsholits and Manabu Tsukada},
  url           = {https://liyun0607.github.io/},
  doi           = {10.1109/IROS60139.2025.11247608},
  year          = {2025},
  date          = {2025-10-19},
  urldate       = {2025-10-19},
  booktitle     = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025)},
  address       = {Hangzhou, China},
  abstract      = {This paper introduces Multi-PrefDrive, a framework that significantly enhances LLM-based autonomous driving through multidimensional preference tuning. Aligning LLMs with human driving preferences is crucial yet challenging, as driving scenarios involve complex decisions where multiple incorrect actions can correspond to a single correct choice. Traditional binary preference tuning fails to capture this complexity. Our approach pairs each chosen action with multiple rejected alternatives, better reflecting real-world driving decisions. By implementing the Plackett-Luce preference model, we enable nuanced ranking of actions across the spectrum of possible errors. Experiments in the CARLA simulator demonstrate that our algorithm achieves an 11.0% improvement in overall score and an 83.6% reduction in infrastructure collisions, while showing perfect compliance with traffic signals in certain environments. Comparative analysis against DPO and its variants reveals that Multi-PrefDrive’s superior discrimination between chosen and rejected actions, which achieving a margin value of 25, and such ability has been directly translates to enhanced driving performance. We implement memory-efficient techniques including LoRA and 4-bit quantization to enable deployment on consumer-grade hardware and will open-source our training code and multi-rejected dataset to advance research in LLM-based autonomous driving systems. Project Page (https://liyun0607.github.io/)},
  keywords      = {CRONOS, ITS},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}