Tsukada Lab. Computer Network Group
← Publications Int'l conference

co-Sound: An interactive medium with WebAR and spatial synchronization

Kazuma Inokuchi, Manabu Tsukada, Hiroshi Esaki

19th IFIP International Conference on Entertainment Computing (ICEC 2020), Xi'an, China, pp. 255–263, Nov 10, 2020

Summary in three lines

  • co-Sound is a browser-based augmented reality player for object-based recordings of music events.
  • It synchronizes AR objects across devices so multiple viewers can interact in the same virtual space.
  • The authors measured synchronization latency and collected positive feedback in a questionnaire with 25 participants.

Figures from the paper

Figure 1: Design and implementation of co-Sound
Figure 1. Design and implementation of co-SoundSource: Figure 1 from Inokuchi et al., IFIP ICEC 2020 (original figure from the authors’ manuscript)The system diagram shows the media and interaction paths that support shared AR playback.
Figure 2: co-Sound playback screens
Figure 2. co-Sound playback screensSource: Figure 2 from Inokuchi et al., IFIP ICEC 2020 (original figure from the authors’ manuscript)The examples show how musical objects are overlaid on the camera view and presented for interaction.

Abstract

An Internet-based media service platform can control recording processes and manage video and audio data interconnected by an IP network. Furthermore, the design and implementation of an object-based system for recording enable the flexible playback of the viewing contents. Augmented Reality (AR) is a three-dimensional video projection technology that allows us to interact with both elements in real space and digital space information. However, there are few examples of its use as a method for audio-visual media platforms. In this study, we propose co-Sound, which is an interactive audio-visual playback application for music events, using WebAR. co-Sound was designed as a multimodal interface that dynamically renders object-based AR in response to various actions from viewers on a web browser with low entry costs. Furthermore, by sharing AR objects among multiple devices in real time and bidirectionally, the relationship between users and contents was extended, and interaction among multiple users in the same AR space was possible. We implemented a prototype application, measured the performance of the AR spatial synchronization, and conducted a questionnaire-based evaluation. For subjective evaluation, 25 people experienced co-Sound and completed a questionnaire. We confirmed that the system was developed by object-based method with AR, achieved low-latency synchronization to accept operations from multiple users in real time, and the general acceptance of the system was very high.

Video

How to cite

K. Inokuchi, M. Tsukada, and H. Esaki, “co-Sound: An interactive medium with WebAR and spatial synchronization,” in 19th IFIP International Conference on Entertainment Computing (ICEC 2020), Xi'an, China, 2020, doi: 10.1007/978-3-030-65736-9_22.

BibTeX

@inproceedings{Inokuchi2020b,
  title         = {co-Sound: An interactive medium with WebAR and spatial synchronization},
  author        = {Kazuma Inokuchi and Manabu Tsukada and Hiroshi Esaki},
  url           = {https://hal.archives-ouvertes.fr/hal-02942505/document?.pdf https://youtu.be/Bn1yPlbgqaA},
  doi           = {10.1007/978-3-030-65736-9_22},
  year          = {2020},
  date          = {2020-11-10},
  booktitle     = {19th IFIP International Conference on Entertainment Computing (ICEC 2020)},
  pages         = {255--263},
  publisher     = {Springer, Cham},
  address       = {Xi'an, China},
  abstract      = {An Internet-based media service platform can control recording processes and manage video and audio data interconnected by an IP network. Furthermore, the design and implementation of an object-based system for recording enable the flexible playback of the viewing contents. Augmented Reality (AR) is a three-dimensional video projection technology that allows us to interact with both elements in real space and digital space information. However, there are few examples of its use as a method for audio-visual media platforms. In this study, we propose co-Sound, which is an interactive audio-visual playback application for music events, using WebAR. co-Sound was designed as a multimodal interface that dynamically renders object-based AR in response to various actions from viewers on a web browser with low entry costs. Furthermore, by sharing AR objects among multiple devices in real time and bidirectionally, the relationship between users and contents was extended, and interaction among multiple users in the same AR space was possible. We implemented a prototype application, measured the performance of the AR spatial synchronization, and conducted a questionnaire-based evaluation. For subjective evaluation, 25 people experienced co-Sound and completed a questionnaire. We confirmed that the system was developed by object-based method with AR, achieved low-latency synchronization to accept operations from multiple users in real time, and the general acceptance of the system was very high.},
  keywords      = {SDM},
  pubstate      = {published},
  tppubtype     = {inproceedings}
}