Tsukada Lab. Computer Network Group

A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions

Rui Zhao, Yun Li, Yuze Fan, Fei Gao, Manabu Tsukada, Zhenhai Gao

IEEE Transactions on Intelligent Transportation Systems, vol. 25, pp. 19365–19398, Sep 18, 2024

Summary in three lines

  • This survey maps deep reinforcement learning approaches to the characteristics of autonomous driving tasks.
  • It identifies six integration modes spanning perception, decision-making, control, and verification and validation.
  • It discusses eight practical issues, including partial observability, reward design, and model training.

Figures from the paper

Figure 2: Organization of the survey
Figure 2. Organization of the surveySource: Figure 2 from Zhao et al., IEEE Transactions on Intelligent Transportation Systems, 2024 (publisher version, © 2024 IEEE)The outline connects deep reinforcement learning methods, autonomous-driving applications, and open challenges.
Figure 8: Ways to apply DRL to autonomous driving
Figure 8. Ways to apply DRL to autonomous drivingSource: Figure 8 from Zhao et al., IEEE Transactions on Intelligent Transportation Systems, 2024 (publisher version, © 2024 IEEE)The figure distinguishes different points at which learned policies enter the driving system.
Figure 10: Challenges and future directions
Figure 10. Challenges and future directionsSource: Figure 10 from Zhao et al., IEEE Transactions on Intelligent Transportation Systems, 2024 (publisher version, © 2024 IEEE)The survey groups the open issues for autonomous vehicles with directions for further research.

Abstract

Autonomous driving (AD) endows vehicles with the capability to drive partly or entirely without human intervention. AD agents generate driving policies based on online perception results, which are crucial to the realization of safe, efficient, and comfortable driving behaviors, particularly in high-dimensional and stochastic traffic scenarios. Currently, deep reinforcement learning (DRL) techniques to derive and validate AD policies have witnessed vast research efforts and have shown rapid development in recent years. However, a comprehensive interpretation and evaluation of their strengths and limitations concerning the full-stack AD tasks remain uncharted. This paper presents a survey of this body of work, which is conducted at three levels. First, it analyzes the multi-level AD task characteristics and delves deeply into the current DRL methodologies primarily employed in AD. Second, a taxonomy of the literature studies is constructed from the system perspective, identifying six modes of DRL model integration into an AD architecture that span the entire spectrum of AD policy processes, from perception understanding and decision-making to motion control, as well as verification and validation. Each literature review comprehensively encompasses the main elements of designing such a system, including modeling partially observable environments, state and action spaces, reward structuring, and the design and training methodologies of neural network models. Finally, an in-depth foresight is conducted on how the eight critical issues of AD application development are addressed by the DRL models tailored for real-world AD challenges.

How to cite

R. Zhao, Y. Li, Y. Fan, F. Gao, M. Tsukada, and Z. Gao, “A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, pp. 19365–19398, 2024, doi: 10.1109/TITS.2024.3452480.

BibTeX

@article{Zhao2024DRLSurvey,
  title         = {A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions},
  author        = {Rui Zhao and Yun Li and Yuze Fan and Fei Gao and Manabu Tsukada and Zhenhai Gao},
  doi           = {10.1109/TITS.2024.3452480},
  isbn          = {1524-9050},
  year          = {2024},
  date          = {2024-09-18},
  urldate       = {2024-09-18},
  journal       = {IEEE Transactions on Intelligent Transportation Systems},
  volume        = {25},
  issue         = {2},
  pages         = {19365--19398},
  abstract      = {Autonomous driving (AD) endows vehicles with the capability to drive partly or entirely without human intervention. AD agents generate driving policies based on online perception results, which are crucial to the realization of safe, efficient, and comfortable driving behaviors, particularly in high-dimensional and stochastic traffic scenarios. Currently, deep reinforcement learning (DRL) techniques to derive and validate AD policies have witnessed vast research efforts and have shown rapid development in recent years. However, a comprehensive interpretation and evaluation of their strengths and limitations concerning the full-stack AD tasks remain uncharted. This paper presents a survey of this body of work, which is conducted at three levels. First, it analyzes the multi-level AD task characteristics and delves deeply into the current DRL methodologies primarily employed in AD. Second, a taxonomy of the literature studies is constructed from the system perspective, identifying six modes of DRL model integration into an AD architecture that span the entire spectrum of AD policy processes, from perception understanding and decision-making to motion control, as well as verification and validation. Each literature review comprehensively encompasses the main elements of designing such a system, including modeling partially observable environments, state and action spaces, reward structuring, and the design and training methodologies of neural network models. Finally, an in-depth foresight is conducted on how the eight critical issues of AD application development are addressed by the DRL models tailored for real-world AD challenges.},
  keywords      = {ITS},
  pubstate      = {published},
  tppubtype     = {article}
}