Tsukada Lab. Computer Network Group
← Publications Int'l conference

A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving

Hanlin Wu, Pengfei Lin, Ehsan Javanmardi, Naren Bao, Bo Qian, Hao Si, Manabu Tsukada

IEEE International Conference on Robotics & Automation (ICRA 2026), Vienna, Austria, Jun 1, 2026

Summary in three lines

  • Collaborative perception research has focused on 3D bounding boxes, and no dataset existed for collaborative semantic occupancy prediction.
  • Implements a high-resolution semantic voxel sensor in CARLA and releases the Co3SOP dataset, built on the V2V scenarios of OPV2V.
  • The Co3SOP-Base baseline fuses features across agents with spatial alignment and attention; collaboration always helps, and the gain grows as the prediction range widens.

Figures from the paper

Figure 1: Annotation pipeline for 3D semantic voxels with a custom sensor in CARLA: voxel configuration, top-down box tracing, object-based occupancy completion and label assignment.
Figure 1. Annotation pipeline for 3D semantic voxels with a custom sensor in CARLA: voxel configuration, top-down box tracing, object-based occupancy completion and label assignment.Source: Figure 1 from Wu et al., IEEE International Conference on Robotics & Automation (ICRA 2026), 2026 (from the authors' manuscript)The dataset's dense ground truth is generated automatically in simulation, step by step, rather than labelled by hand; this is how Co3SOP obtains complete voxel labels including occluded areas.
Figure 3: The Co3SOP-Base baseline: (1) the ego-vehicle prediction pipeline from multi-camera images; (2) fusion of voxel features shared by connected vehicles with 3D affine transformation and masked deformable attention.
Figure 3. The Co3SOP-Base baseline: (1) the ego-vehicle prediction pipeline from multi-camera images; (2) fusion of voxel features shared by connected vehicles with 3D affine transformation and masked deformable attention.Source: Figure 3 from Wu et al., IEEE International Conference on Robotics & Automation (ICRA 2026), 2026 (from the authors' manuscript)Each vehicle predicts occupancy from its own cameras, then aligns and merges voxel features received from nearby vehicles, so the prediction covers areas the ego vehicle cannot see.
Figure 4: Ablation on the number of collaborating vehicles used for feature fusion, at three prediction ranges.
Figure 4. Ablation on the number of collaborating vehicles used for feature fusion, at three prediction ranges.Source: Figure 4 from Wu et al., IEEE International Conference on Robotics & Automation (ICRA 2026), 2026 (from the authors' manuscript)Accuracy (mIoU) rises most with the first one or two collaborating vehicles and then levels off; the gain is larger at the longer ranges of 51.2 m and 76.8 m.

Abstract

3D semantic occupancy prediction is an emerging perception paradigm in autonomous driving, providing a voxel-level representation of both geometric details and semantic categories. However, its effectiveness is inherently constrained in single-vehicle setups by occlusions, restricted sensor range, and narrow viewpoints. To address these limitations, collaborative perception enables the exchange of complementary information, thereby enhancing the completeness and accuracy of predictions. Despite its potential, research on collaborative 3D semantic occupancy prediction is hindered by the lack of dedicated datasets. To bridge this gap, we design a high-resolution semantic voxel sensor in CARLA to produce dense and comprehensive annotations. We further develop a baseline model that performs inter-agent feature fusion via spatial alignment and attention aggregation. In addition, we establish benchmarks with varying prediction ranges designed to systematically assess the impact of spatial extent on collaborative prediction. Experimental results demonstrate the superior performance of our baseline, with increasing gains observed as range expands.

Video

Code · Data

Related research and software

Related news

How to cite

H. Wu, P. Lin, E. Javanmardi, N. Bao, B. Qian, H. Si, and M. Tsukada, “A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving,” in IEEE International Conference on Robotics & Automation (ICRA 2026), Vienna, Austria, 2026.

BibTeX

@inproceedings{Wu2026,
  title         = {A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving},
  author        = {Hanlin Wu and Pengfei Lin and Ehsan Javanmardi and Naren Bao and Bo Qian and Hao Si and Manabu Tsukada},
  url           = {https://arxiv.org/abs/2506.17004 https://github.com/tlab-wide/Co3SOP https://youtu.be/VlrqcdkMyn8},
  eprint        = {2506.17004},
  archiveprefix = {arXiv},
  primaryclass  = {cs.CV},
  year          = {2026},
  date          = {2026-06-01},
  urldate       = {2026-06-01},
  booktitle     = {IEEE International Conference on Robotics & Automation (ICRA 2026)},
  address       = {Vienna, Austria},
  abstract      = {3D semantic occupancy prediction is an emerging perception paradigm in autonomous driving, providing a voxel-level representation of both geometric details and semantic categories. However, its effectiveness is inherently constrained in single-vehicle setups by occlusions, restricted sensor range, and narrow viewpoints. To address these limitations, collaborative perception enables the exchange of complementary information, thereby enhancing the completeness and accuracy of predictions. Despite its potential, research on collaborative 3D semantic occupancy prediction is hindered by the lack of dedicated datasets. To bridge this gap, we design a high-resolution semantic voxel sensor in CARLA to produce dense and comprehensive annotations. We further develop a baseline model that performs inter-agent feature fusion via spatial alignment and attention aggregation. In addition, we establish benchmarks with varying prediction ranges designed to systematically assess the impact of spatial extent on collaborative prediction.  Experimental results demonstrate the superior performance of our baseline, with increasing gains observed as range expands.},
  keywords      = {CRONOS, ITS, Software-Data},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}