Multi-modal pose estimation in XR applications leveraging integrated sensing and communication

Nabeel Nisar Bhat, Javad Sameri, Jakob Struye, Maria Torres Vega, Rafael Berkvens, Jeroen Famaey
University of Antwerp-imec
ACM MobiCom Workshop 2023
Paper Supplementary Code arXiv
-->

Abstract

Mobile extended reality (XR) applications are anticipated to generate substantial traffic for 6G. Such applications not only require high data rate and low-latency transmissions, but also accurate and real-time pose estimation to enable interactive and immersive experiences. While sub-6 GHz signals have been exploited for pose estimation, they cannot cope up with multi-gigabit data rates required by XR applications. Instead, mobile communications at mmWave frequencies can potentially support data rates up to several giga-bits per second (Gbps) and, therefore, can be used to deliver XR content wirelessly to the Head-Mounted Display (HMD). Moreover, mmWave frequencies can offer improved sensing due to the large available bandwidth. Therefore, mmWave communications can play a crucial role in enabling device-free interactivity by offering both high-speed communication and accurate sensing capabilities. However, mmWave propagation characteristics are different from sub-6 GHz. Path loss plays a significant role, and can lead to degraded sensing performance. Therefore, our proposal supplements wireless sensing at mmWave frequencies with wireless electromyography (EMG) armbands. By capturing patterns of muscle activities, we can counteract the limitations of mmWave-based pose estimation, thereby enriching the granularity and precision of pose estimation. This paper proposes a conceptual architecture to achieve multi-modal pose estimation for XR applications. Early results highlight the shortcomings of mmWave-based sensing, and we identify future steps and opportunities on integration of both approaches.

Key Takeaways

  • Human pose estimation using commercial-off-the-shelf mmWave Wi-Fi Access Points.
  • Wireless signals lack visual cues. Complex model-based approaches and signal processing are needed to estimate poses.
  • Instead, the paper proposes a multi-modal setup leveraging Vision-based and Wi-Fi signals.
  • A DNN is trained to map wireless signals from signal space to visual skeletal representations, using poses captured by Kinect as ground truth.
  • At test time, skeletons are predicted solely based on CSI from mmWave COTS Wi-Fi.
  • Low error between predicted skeletons and the ground-truth.

BibTeX

@inproceedings{bhat2023multi,
  title={Multi-modal pose estimation in XR applications leveraging integrated sensing and communication},
  author={Bhat, Nabeel Nisar and Sameri, Javad and Struye, Jakob and Vega, Maria Torres and Berkvens, Rafael and Famaey, Jeroen},
  booktitle={Proceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems},
  pages={261--267},
  year={2023}
}