Tsukada Lab. Computer Network Group

Cross-Attention Enhanced End-to-End Autonomous Driving in Unprotected Turns

Dongyang Li

Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo, Mar 31, 2025

Summary in three lines

  • The thesis studies end-to-end driving at unprotected turns, where actions depend on both the view and vehicle motion.
  • It fuses image and ego-state features with cross-attention in an imitation-learning policy.
  • In an AWSIM scenario with unprotected turns, the method outperformed the reported imitation-learning baseline.

Figures from the paper

Figure 4.1: Overview of the cross-attention enhanced imitation-learning framework for unprotected turns
Figure 4.1. Overview of the cross-attention enhanced imitation-learning framework for unprotected turnsSource: Figure 4.1 from Master's thesis, The University of Tokyo, 2025 (author's version)A front camera image becomes patch tokens through a pretrained ViT, while heading, position, turn command and speed become a state token. Cross-attention weights the image patches, and the policy outputs an Ackermann control command.
Figure 5.5: Success rate of Autoware and three learned policies in the AWSIM unprotected right-turn scenario
Figure 5.5. Success rate of Autoware and three learned policies in the AWSIM unprotected right-turn scenarioSource: Figure 5.5 from Master's thesis, The University of Tokyo, 2025 (author's version)The cross-attention policy reached a 94% success rate, compared with 84% for CIL-ViT, 70% for ViT-Only and 42% for Autoware.

Abstract

Performing an unprotected turn at the intersection is a complex scenario for autonomous vehicles. It requires a comprehensive understanding of the surrounding environment, while also highly relying on the current state of the ego vehicle to make safer decisions. A conventional way to learn end-to-end autonomous driving is imitation learning, which stands for learning from expert demonstrations. While most imitation learning methods focus on imitating the expert action, they often fail to imitate a complex policy efficiently when the ego vehicle’s states are crucial to the scenario because there might be arbitrary optimal actions under different states. To address this issue and investigate how vehicle states affect autonomous driving, we present a novel cross-attention enhanced imitation learning approach for end-to-end autonomous driving in unprotected turns, focusing on capturing the relationships between the ego vehicle’s states and its perception of the environment. We evaluated our model in the AWSIM simulator, an open source autonomous driving simulator, and the results demonstrate that our model outperformed conventional imitation learning-based baselines in performing unprotected turn scenarios, showcasing its ability to imitate a complex policy efficiently.

How to cite

D. Li, “Cross-Attention Enhanced End-to-End Autonomous Driving in Unprotected Turns,” in Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo, 2025.

BibTeX

@mastersthesis{Li2025,
  title         = {Cross-Attention Enhanced End-to-End Autonomous Driving in Unprotected Turns},
  author        = {Dongyang Li},
  year          = {2025},
  date          = {2025-03-31},
  urldate       = {2025-03-31},
  school        = {Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo},
  abstract      = {Performing an unprotected turn at the intersection is a complex scenario for autonomous vehicles. It requires a comprehensive understanding of the surrounding environment, while also highly relying on the current state of the ego vehicle to make safer decisions. A conventional way to learn end-to-end autonomous driving is imitation learning, which stands for learning from expert demonstrations. While most imitation learning methods focus on imitating the expert action, they often fail to imitate a complex policy efficiently when the ego vehicle’s states are crucial to the scenario because there might be arbitrary optimal actions under different states. To address this issue and investigate how vehicle states affect autonomous driving, we present a novel cross-attention enhanced imitation learning approach for end-to-end autonomous driving in unprotected turns, focusing on capturing the relationships between the ego vehicle’s states and its perception of the environment. We evaluated our model in the AWSIM simulator, an open source autonomous driving simulator, and the results demonstrate that our model outperformed conventional imitation learning-based baselines in performing unprotected turn scenarios, showcasing its ability to imitate a complex policy efficiently.},
  keywords      = {Thesis},
  pubstate      = {published},
  tppubtype     = {mastersthesis}
}