Tsukada Lab. Computer Network Group

Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing

Alex Orsholits, Manabu Tsukada

39th International Conference on Advanced Information Networking and Applications (AINA 2025), Barcelona, Spain, 2025年4月9日

3行要約

  • エッジAI処理で複合現実ヘッドセットを補い、周囲の環境に応じたやり取りを可能にする。
  • ヘッドセット、エッジアクセラレータ、クラウドの言語・画像モデルが連携して複数の入力を処理する。
  • 端末間で空間認識を分担し、得られた文脈を利用者の表示に戻す構成を示す。

アブストラクト(原文)

Spatial computing is evolving towards leveraging data streaming for computationally demanding applications, facilitating a shift to lightweight, untethered, and standalone devices. These devices are therefore ideal candidates for co-processing, where real-time context understanding and low-latency data streaming are fundamental for seamless, general-purpose Mixed Reality (MR) experiences. This paper demonstrates and evaluates a scalable approach to augmented contextual understanding in MR by implementing multi-modal edge AI co-processing through a Hailo-8 AI accelerator, a low-power ARM-based single board computer (SBC), and the Magic Leap 2 AR headset. The proposed system utilises the native WebRTC streaming capabilities of the Magic Leap 2 to continuously stream camera data to the edge co-processor, where a collection of vision AI models-object detection, pose estimation, face recognition, and depth estimation-are executed. The resulting inferences are then streamed back to the headset for spatial re-projection and transmitted to cloud-based systems for further integration with large-scale AI models, such as LLMs and VLMs. This seamless integration enhances real-time contextual understanding in MR while facilitating advanced multi-modal, multi-device collaboration, supporting richer, scalable spatial cognition across distributed systems.

関連する研究・ソフトウェア

引用方法

A. Orsholits and M. Tsukada, “Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing,” in 39th International Conference on Advanced Information Networking and Applications (AINA 2025), Barcelona, Spain, 2025, doi: 10.1007/978-3-031-87772-8_3.

BibTeX

@inproceedings{Orsholits2025,
  title         = {Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing},
  author        = {Alex Orsholits and Manabu Tsukada},
  url           = {https://link.springer.com/chapter/10.1007/978-3-031-87772-8_3},
  doi           = {10.1007/978-3-031-87772-8_3},
  isbn          = {978-3-031-87771-1},
  year          = {2025},
  date          = {2025-04-09},
  urldate       = {2025-04-09},
  booktitle     = {39th International Conference on Advanced Information Networking and Applications (AINA 2025)},
  address       = {Barcelona, Spain},
  abstract      = {Spatial computing is evolving towards leveraging data streaming for computationally demanding applications, facilitating a shift to lightweight, untethered, and standalone devices. These devices are therefore ideal candidates for co-processing, where real-time context understanding and low-latency data streaming are fundamental for seamless, general-purpose Mixed Reality (MR) experiences. This paper demonstrates and evaluates a scalable approach to augmented contextual understanding in MR by implementing multi-modal edge AI co-processing through a Hailo-8 AI accelerator, a low-power ARM-based single board computer (SBC), and the Magic Leap 2 AR headset. The proposed system utilises the native WebRTC streaming capabilities of the Magic Leap 2 to continuously stream camera data to the edge co-processor, where a collection of vision AI models-object detection, pose estimation, face recognition, and depth estimation-are executed. The resulting inferences are then streamed back to the headset for spatial re-projection and transmitted to cloud-based systems for further integration with large-scale AI models, such as LLMs and VLMs. This seamless integration enhances real-time contextual understanding in MR while facilitating advanced multi-modal, multi-device collaboration, supporting richer, scalable spatial cognition across distributed systems.},
  keywords      = {SDM},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}