Tsukada Lab. Computer Network Group
← Publications Int'l conference

Design of Digital Twin Architecture for 3D Audio Visualization in AR

Tokio Takada, Jin Nakazato, Alex Orsholits, Manabu Tsukada, Hideya Ochiai, Hiroshi Esaki

The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024), Hong Kong, China, Aug 12, 2024

Summary in three lines

  • The study designs a digital twin for spatial representations of music.
  • Recorded audio is mapped into physical space using an AR device and visual positioning.
  • The prototype is evaluated through positioning and sound localization measurements and a user study.

Figures from the paper

Figure 2: Digital twin architecture for AR audio
Figure 2. Digital twin architecture for AR audioSource: Figure 2 from Takada et al., The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024), 2024 (original artwork from the authors’ manuscript)The architecture connects the representation of physical locations with the service that delivers spatial audio to the AR client.

Abstract

Digital twins have recently attracted attention from academia and industry as a technology connecting physical space and cyberspace. Digital twins are compatible with Augmented Reality (AR) and Virtual Reality (VR), enabling us to understand information in cyberspace. In this study, we focus on music and design an architecture for a 3D representation of music using a digital twin. Specifically, we organize the requirements for a digital twin for music and design the architecture. We establish a method to perform 3D representation in cyberspace and map the recorded audio data in physical space. In this paper, we implemented the physical space representation using a smartphone as an AR device and employed a visual positioning system (VPS) for self-positioning. For evaluation, in addition to system errors in the 3D representation of audio data, we conducted a questionnaire evaluation with several users as a user study. From these results, we evaluated the effectiveness of the implemented system. At the same time, we also found issues we need to improve in the implemented system in future works.

Video

Related video: Integrated Spatial Information Platform through BIM-Based Digital Twin

Related research and software

Related news

How to cite

T. Takada, J. Nakazato, A. Orsholits, M. Tsukada, H. Ochiai, and H. Esaki, “Design of Digital Twin Architecture for 3D Audio Visualization in AR,” in The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024), Hong Kong, China, 2024, doi: 10.1109/MetaCom62920.2024.00044.

BibTeX

@inproceedings{Takada2024,
  title         = {Design of Digital Twin Architecture for 3D Audio Visualization in AR},
  author        = {Tokio Takada and Jin Nakazato and Alex Orsholits and Manabu Tsukada and Hideya Ochiai and Hiroshi Esaki},
  doi           = {10.1109/MetaCom62920.2024.00044},
  year          = {2024},
  date          = {2024-08-12},
  urldate       = {2024-08-12},
  booktitle     = {The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024)},
  address       = {Hong Kong, China},
  abstract      = {Digital twins have recently attracted attention from academia and industry as a technology connecting physical space and cyberspace. Digital twins are compatible with Augmented Reality (AR) and Virtual Reality (VR), enabling us to understand information in cyberspace. In this study, we focus on music and design an architecture for a 3D representation of music using a digital twin. Specifically, we organize the requirements for a digital twin for music and design the architecture. We establish a method to perform 3D representation in cyberspace and map the recorded audio data in physical space. In this paper, we implemented the physical space representation using a smartphone as an AR device and employed a visual positioning system (VPS) for self-positioning. For evaluation, in addition to system errors in the 3D representation of audio data, we conducted a questionnaire evaluation with several users as a user study. From these results, we evaluated the effectiveness of the implemented system. At the same time, we also found issues we need to improve in the implemented system in future works.},
  key           = {CREST},
  keywords      = {SDM},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}