Tsukada Lab. Computer Network Group
← Publications Int'l conference

Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing

Alex Orsholits, Manabu Tsukada

39th International Conference on Advanced Information Networking and Applications (AINA 2025), Barcelona, Spain, Apr 9, 2025

Summary in three lines

  • The work augments mixed-reality headsets with edge AI processing for richer interaction with their surroundings.
  • A headset, an edge accelerator and cloud language or vision models collaborate to process multimodal inputs.
  • The architecture supports distributing spatial perception across devices and returning context to the user’s display.

Abstract

Spatial computing is evolving towards leveraging data streaming for computationally demanding applications, facilitating a shift to lightweight, untethered, and standalone devices. These devices are therefore ideal candidates for co-processing, where real-time context understanding and low-latency data streaming are fundamental for seamless, general-purpose Mixed Reality (MR) experiences. This paper demonstrates and evaluates a scalable approach to augmented contextual understanding in MR by implementing multi-modal edge AI co-processing through a Hailo-8 AI accelerator, a low-power ARM-based single board computer (SBC), and the Magic Leap 2 AR headset. The proposed system utilises the native WebRTC streaming capabilities of the Magic Leap 2 to continuously stream camera data to the edge co-processor, where a collection of vision AI models-object detection, pose estimation, face recognition, and depth estimation-are executed. The resulting inferences are then streamed back to the headset for spatial re-projection and transmitted to cloud-based systems for further integration with large-scale AI models, such as LLMs and VLMs. This seamless integration enhances real-time contextual understanding in MR while facilitating advanced multi-modal, multi-device collaboration, supporting richer, scalable spatial cognition across distributed systems.

Related research and software

How to cite

A. Orsholits and M. Tsukada, “Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing,” in 39th International Conference on Advanced Information Networking and Applications (AINA 2025), Barcelona, Spain, 2025, doi: 10.1007/978-3-031-87772-8_3.

BibTeX

@inproceedings{Orsholits2025,
  title         = {Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing},
  author        = {Alex Orsholits and Manabu Tsukada},
  url           = {https://link.springer.com/chapter/10.1007/978-3-031-87772-8_3},
  doi           = {10.1007/978-3-031-87772-8_3},
  isbn          = {978-3-031-87771-1},
  year          = {2025},
  date          = {2025-04-09},
  urldate       = {2025-04-09},
  booktitle     = {39th International Conference on Advanced Information Networking and Applications (AINA 2025)},
  address       = {Barcelona, Spain},
  abstract      = {Spatial computing is evolving towards leveraging data streaming for computationally demanding applications, facilitating a shift to lightweight, untethered, and standalone devices. These devices are therefore ideal candidates for co-processing, where real-time context understanding and low-latency data streaming are fundamental for seamless, general-purpose Mixed Reality (MR) experiences. This paper demonstrates and evaluates a scalable approach to augmented contextual understanding in MR by implementing multi-modal edge AI co-processing through a Hailo-8 AI accelerator, a low-power ARM-based single board computer (SBC), and the Magic Leap 2 AR headset. The proposed system utilises the native WebRTC streaming capabilities of the Magic Leap 2 to continuously stream camera data to the edge co-processor, where a collection of vision AI models-object detection, pose estimation, face recognition, and depth estimation-are executed. The resulting inferences are then streamed back to the headset for spatial re-projection and transmitted to cloud-based systems for further integration with large-scale AI models, such as LLMs and VLMs. This seamless integration enhances real-time contextual understanding in MR while facilitating advanced multi-modal, multi-device collaboration, supporting richer, scalable spatial cognition across distributed systems.},
  keywords      = {SDM},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}