Summary

Real-Time Markerless Motion Capture explores how machine learning can remove the need for physical markers in location-based virtual reality (LBVR) while preserving the speed and accuracy required for immersive multi-user experiences.

Traditional LBVR motion capture systems rely on active or passive infrared markers attached to users. While this provides stable and low-latency tracking, preparing participants with markers is time-consuming and creates additional maintenance and operational constraints.

The project develops a markerless pipeline optimized for real-time performance. Camera images are processed at 60 frames per second, 2D body poses are estimated independently in each view, and the results are combined into full 3D skeletons. To meet the strict latency requirements of immersive VR, the pipeline is optimized across modern CPU and GPU architectures, balancing tracking accuracy with computational speed.

A second major challenge is the creation of sufficiently large and accurately annotated datasets for training machine-learning models. Rather than relying only on existing datasets, Artanim developed its Synthetic Factory to generate diverse virtual populations with variations in body shape, age, appearance, clothing, footwear, hairstyles and movement. Because the underlying virtual skeleton is known exactly, these synthetic subjects provide reliable ground-truth data for training and evaluation.

The project was developed in collaboration with Vicon. The first results were publicly demonstrated at SIGGRAPH 2023, where six participants were tracked simultaneously in real time without markers during The Clockwork Forest, a Dreamscape location-based VR experience created in partnership with Audemars Piguet. The showcase received a CGW Silver Edge Award for technological innovation.

Partners

Artanim
Definition of VR requirements, evaluation of machine-learning-based tracking algorithms, development of the Synthetic Factory, and implementation and fine-tuning of the multimodal tracking solution.

Vicon
Hardware development, implementation of machine-learning-based tracking algorithms, and real-time pose solving.

Related Publications

Nagorny P, Kevelham B, Chagué S, Charbonnier C. A Comprehensive Review of Real-Time Multi-View Multi-Person Markerless Motion Capture, ACM Comput Surv, 58(3):1–34, 2025.
PDF · DOI