LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training
Best Paper Award (Bronze)International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024), Tokyo, Japan, pp. 1–7, Nov 28, 2024
arXivDOI: 10.1145/3732437.3732763
Summary in three lines
- The method fuses LiDAR and image features for panoptic segmentation.
- Two model changes extend the fusion approach to video without training on video examples.
- Image and video panoptic metrics improve by up to five points.
Figures from the paper

Abstract
Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied to cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS). This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.
Related news
How to cite
F. Ayar, E. Javanmardi, M. Tsukada, M. Javanmardi, and M. Rahmati, “LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training,” in International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024), Tokyo, Japan, 2024, doi: 10.1145/3732437.3732763.
BibTeX
@inproceedings{Ayar2024,
title = {LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training},
author = {Fardin Ayar and Ehsan Javanmardi and Manabu Tsukada and Mahdi Javanmardi and Mohammad Rahmati},
url = {https://arxiv.org/abs/2412.20881},
doi = {10.1145/3732437.3732763},
year = {2024},
date = {2024-11-28},
urldate = {2024-11-28},
booktitle = {International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024)},
pages = {1--7},
address = {Tokyo, Japan},
abstract = {Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied to cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS). This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.},
note = {Best Paper Award (Bronze)},
keywords = {Award, ITS},
pubstate = {published},
tppubtype = {inproceedings}
}