Tsukada Lab. Computer Network Group
← Publications Int'l conference

LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training

Best Paper Award (Bronze)

Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi, Mohammad Rahmati

International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024), Tokyo, Japan, pp. 1–7, Nov 28, 2024

Summary in three lines

  • The method fuses LiDAR and image features for panoptic segmentation.
  • Two model changes extend the fusion approach to video without training on video examples.
  • Image and video panoptic metrics improve by up to five points.

Figures from the paper

Figure 2: LiDAR and camera feature fusion for panoptic segmentation
Figure 2. LiDAR and camera feature fusion for panoptic segmentationSource: Figure 2 from Ayar et al., International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024), 2024 (arXiv:2412.20881, authors’ version)The highlighted modules feed LiDAR information into an image-based model to improve segmentation.

Abstract

Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied to cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS). This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.

Related news

How to cite

F. Ayar, E. Javanmardi, M. Tsukada, M. Javanmardi, and M. Rahmati, “LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training,” in International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024), Tokyo, Japan, 2024, doi: 10.1145/3732437.3732763.

BibTeX

@inproceedings{Ayar2024,
  title         = {LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training},
  author        = {Fardin Ayar and Ehsan Javanmardi and Manabu Tsukada and Mahdi Javanmardi and Mohammad Rahmati},
  url           = {https://arxiv.org/abs/2412.20881},
  doi           = {10.1145/3732437.3732763},
  year          = {2024},
  date          = {2024-11-28},
  urldate       = {2024-11-28},
  booktitle     = {International Conference on Intelligent Computing and its Emerging Applications (ICEA 2024)},
  pages         = {1--7},
  address       = {Tokyo, Japan},
  abstract      = {Panoptic segmentation, which combines instance and semantic segmentation, has gained a lot of attention in autonomous vehicles, due to its comprehensive representation of the scene. This task can be applied to cameras and LiDAR sensors, but there has been a limited focus on combining both sensors to enhance image panoptic segmentation (PS). Although previous research has acknowledged the benefit of 3D data on camera-based scene perception, no specific study has explored the influence of 3D data on image and video panoptic segmentation (VPS). This work seeks to introduce a feature fusion module that enhances PS and VPS by fusing LiDAR and image data for autonomous vehicles. We also illustrate that, in addition to this fusion, our proposed model, which utilizes two simple modifications, can further deliver even more high-quality VPS without being trained on video data. The results demonstrate a substantial improvement in both the image and video panoptic segmentation evaluation metrics by up to 5 points.},
  note          = {Best Paper Award (Bronze)},
  keywords      = {Award, ITS},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}