0%
This research investigates a co-processing framework for real-time visual inference in mixed reality (MR), targeting the computational limitations of current head-mounted displays. While modern MR devices support spatial tracking, they lack the capacity to perform high-resolution semantic scene analysis onboard. To address this, we propose an architecture in which semantic segmentation, object detection, and classification are offloaded to external edge AI accelerators.
The system integrates commercial MR headsets (e.g. Magic Leap 2, Meta Quest) with compact processing nodes equipped with devices such as the Hailo-8. Visual input—either full-resolution frames or compressed intermediate features—is transmitted over low-latency wireless links. Inference is performed externally, and structured outputs (e.g. object masks, bounding boxes, scene labels) are returned to the headset and rendered in-engine via Unity or Unreal.
This enables applications to interact meaningfully with their physical surroundings. Virtual content can be dynamically anchored to recognized objects, navigation systems can adapt to context-dependent spatial cues, and application logic can respond to real-time environmental classification. Unlike traditional onboard pipelines, this approach decouples perception from headset constraints, allowing for broader model complexity and higher update rates.
Initial deployments include mobile outdoor scenarios with sub-100 ms round-trip latency, as well as indoor annotation tasks requiring dense semantic feedback. Multi-headset use cases have also been validated through shared inference nodes.
By enabling structured, real-time perception at the scene level, this research contributes to the advancement of Spatial AI: the integration of geometric mapping, semantic interpretation, and real-world interactivity within mixed reality systems.
autonomous driving v2x
v2x
digital twins extended reality
digital twins
autonomous driving machine learning
machine learning v2x
autonomous driving v2x
extended reality
