Real Virtuality

Real Virtuality

Real Virtuality

Immersive platform

Project Info

Start date:
April 2015

End date:

Funding:

Coordinator:
Artanim

Summary

Artanim is the inventor and continuously develops since 2015 the VR technology driving Dreamscape Immersive. This multi-user immersive platform combines a 3D environment – which can be seen and heard through a VR headset – with a real-life stage set incorporating haptic feedback elements.

The user’s movements are sparsely captured in real time and translated into a full-body animation thanks to a deep understanding of body mechanics. Contrary to other static position VR systems, the platform allows up to eight users to truly feel immersed in a VR scene. They are rendered as characters (avatars) inside a computer generated world where they can move physically, interact with objects and other players, and truly experience worlds previously accessible only in their imagination. The bodies of the users thus become the interface between the real and virtual worlds.

The platform combines the following features:

  • Wireless: complete freedom of movement across large spaces.
  • Social: interactive multi-user experiences within a shared environment or across connected pods.
  • Accurate: full-body and physical objects tracking with less than 1 millimeter deviation.
  • Real-time: zero discernible lag eliminating most concerns of motion sickness.
  • Flexible: SDK allowing content creators to easily create experiences for this particular platform.

This platform is leveraged by Dreamscape Immersive through their worldwide location-based VR entertainment centers, as well as through educational and training programs and other enterprise solutions.

The platform was also used to produce VR_I, the first ever choreographic work in immersive virtual reality, as well as Geneva 1850, a time traveling experience and historical reconstruction into the Geneva of 1850.

Related Publications

Chagué S, Charbonnier C. Real Virtuality: A Multi-User Immersive Platform Connecting Real and Virtual Worlds, VRIC 2016 Virtual Reality International Conference – Laval Virtual, Laval, France, ACM New York, NY, USA, March 2016.
PDF
 
Chagué S, Charbonnier C. Digital Cloning for an Increased Feeling of Presence in Collaborative Virtual Reality Environments, Proc. of 6th Int. Conf. on 3D Body Scanning Technologies, Lugano, Switzerland, October 2015.
PDF

Charbonnier C, Trouche V. Real Virtuality: Perspective offered by the Combination of Virtual Reality Headsets and Motion Capture, White Paper, August 2015.
PDF

Awards and Recognitions

Presence

Presence

Presence

A toolset for hyper-realistic and XR-based human-human and human-machine interactions

Project Info

Start date:
January 2024

End date:
December 2027

Funding:
EU Commission (GAP-101135025) and SERI (REF-1131-52104)

Coordinator:
Fundació i2CAT

Summary

The concept of presence can be understood as a synthesis of interrelated psychophysical ingredients where multiple perceptual dimensions intervene. A better understanding of how to impact specific aspects such as plausibility, the illusion that virtual events are really happening, co-presence, the illusion of being with others, or place illusion, the feeling of being there, is key to improve the quality of XR experiences. The availability and performance of advanced technologies will let us reach high levels of presence in XR, essential to get us closer than ever to the old VR dream: to be anywhere, doing anything, together with others from any place. The PRESENCE project will impact multiple dimensions of presence in physical-digital worlds, addressing three main challenges: i) how to create realistic visual interactions among remote humans, delivering high-end holoportation paradigms based on live volumetric video capturing, compression and optimization techniques under heterogeneous computation and network conditions; ii) how to provide realistic touch among remote users and synthetic objects, developing novel haptic systems and enabling spatial multi-device synchronization in multi-user scenarios; iii) how to produce realistic social interactions among avatars and agents, generating AI virtual humans, representing real users or AI agents.

The contribution of Artanim will focus on smart autonomous characters, which are able to generate physically-plausible behavior integrating the cues of avatars of human users or other autonomous characters in specific scenarios. Specific parameters will be studied, for the movement to respond plausibly to social cues such as interpersonal differences or to collaboration tasks. Interactive virtual agents may take into account the position of others to respect implicit proxemics rules by generating small step rotations, or if a user needs an object, the interactive virtual agents will grab and offer it.

Partners

Fundació i2Cat (Spain)

Actronika (France)

Universitaet Hamburg (Germany)

Ethniko Kentro Erevnas Kai Tehcnologikis Anapty (Greece)

Raytrix GMBH  (Germany)

SenseGlove B.V. (Netherlands)

Go Touch VR SAS (France)

Didimo, S.A.(Portugal)

Vection Italy Srl (Italy)

Universitat de Barcelona (Spain)

Unity Technologies (Denmark)

Sound Holding B.V. (Netherlands)

Interuniversitair Micro-Electronica Centrum (Belgium)

Joanneum Research Forschungsgesellschaft MBH (Austria)

SyncVR Medical B.V. (Netherlands)

Zaubar UG (Haftungsbeschraenkt) (Germany)

Artanim

MoRehab XR

MoRehab XR

MoRehab XR

Motion tracked rehabilitation through extended realities

Project Info

Start date:
August 2020

End date:
December 2024

Funding:

Coordinator:
Artanim

Summary

After a stabilization surgery or a traumatic accident, physical rehabilitation is often required to restore the joint’s normal function. This process requires very regular and repetitive training sessions. The regularity and involvement of the patient being critical for success of the procedure, maintaining them interested and motivated while repeating similar exercises over and over is one of the main challenges of the physical therapist. During these sessions, it is also difficult to objectively monitor small progresses or the lack of the latter. On the other end of the spectrum, some video games have proved to be very effective in motivating people to perform pretty repetitive tasks on their controllers in order to travel through a story or a challenge.

The goal of this project is to use the captivating capabilities of video games to entertain and motivate the patients through their rehabilitation, by designing a set of “exercise games” (called exergames) that will be won by performing the correct routines required for a proper physical rehabilitation. Through the use of modern motion tracking technology, the exergames will also be able to track the evolution of the physical recovery of the patients on a session per session basis, to adapt the challenges that the patients will face based on their needs and actual capacities, and if need be to alert the medical personnel early if some problems persist, or if no progresses are observed.

In this project, Artanim is in charge of building a platform suitable for these exergames, as well as designing and developing these exergames, and implementing the tools for a standard assessment of the patient’s performance. On the other hand, the medical usability and validity of the setup and exergames will be tested and assessed by the clinical team of La Tour Hospital over a large set of voluntary patients. Both teams will also work together to develop metrics to evaluate and adapt the patient’s performance during the sessions, thanks to the extended capabilities of the motion tracking system with respect to the data available in conventional physical therapy.

Partners

Artanim
Conception of the exergames and of a platform adapted to perform them, development of a set of medical scores adapted to the extended capabilities of the motion tracking system

La Tour Hospital – Division of Orthopedic and Trauma Surgery
Clinical tests, co-design of new physical evaluation metrics adapted to the motion tracking capabilities

Char4VR

Char4VR

Char4VR

Interactive characters for VR experiences

Project Info

Start date:
September 2020

End date:
December 2023

Funding:

Coordinator:
Artanim

Summary

Virtual Reality (VR) opens the possibility to develop a new kind of content format which, on one hand, is experienced as a rich narrative, just as movies or theater plays are, while on the other hand, it allows interaction with autonomous virtual characters. The experience would feel like having different autonomous characters unfold a consistent story plot, just as a movie or a theater play does, within an interactive VR simulation. Players could choose to participate in the events and affect some parts of the story, or just watch how the plot unfolds around them.

The main challenge to realize this vision is that current techniques for interactive character animation are designed for video games, and do not offer the subtle multi-modal interaction that VR users spontaneously expect.  The main goal of this project is to explore different techniques for interactive character animation that help creates interactive characters that can engage in a more compelling way with players. To achieve this goal, we use a combination of research techniques derived from computer graphics, machine learning and cognitive psychology.

Related Publications

Llobera J, Charbonnier C. Physics-based character animation and human motor control, Phys Life Rev, 46:190-219, 2023.

Llobera J, Jacquat V, Calabrese C, Charbonnier C. Playing the mirror game in virtual reality with an autonomous character, Sci Rep, 12:21329, 2022.
PDF

Llobera J, Charbonnier C. Physics-based character animation for Virtual Reality, Open Access Tools and Libraries for Virtual Reality, IEEE VR Workshop, 2022 Best Open Source tool Award, March 2022.
PDF

Llobera J, Booth J, Charbonnier C. Physics-based character animation controllers for videogame and virtual reality production, 14th ACM SIGGRAPH Conference on Motion, Interaction and Games, November 2021.
PDF

Llobera J, Booth J, Charbonnier C. New Techniques on Interactive Character Animation, SIGGRAPH ’21: short course, ACM, New York, NY, USA, August 2021.

Llobera J, Charbonnier C. Interactive Characters for Virtual Reality Stories, ACM International Conference on Interactive Media Experiences (IMX \’21), ACM, New York, NY, USA, June 2021.
PDF

Markerless Mocap

Markerless Mocap

Markerless Mocap

Killing the markers

Project Info

Start date:
September 2020

End date:
September 2023

Funding:

Coordinator:
Artanim

Summary

Current motion capture (Mocap) solutions used in Location-based Virtual Reality (LBVR) experiences rely on a set of either active or passive infrared markers to track the user’s movements within the experience space. While this offers a stable and low-latency tracking result ideally suited for a VR scenario, equipping users with such markers is cumbersome, and the maintenance and cleaning of such devices takes time.

The Markerless Mocap project aims to leverage modern Machine Learning (ML) approaches to create a markerless solution for the capture of users in LBVR scenarios. The low-latency requirements imposed by this scenario are the primary challenge for the project, with an ideal photon-to-skeleton latency being in the order of 50ms or less. To achieve this goal the project focuses on approaches to pose-estimation which strike a good balance between accuracy and speed. The markerless pipeline consists of 3 stages: the processing of raw camera input at 60 frames per second, the 2D pose estimates of subjects in each view, and the final assembly into a full 3D skeleton. To manage this computationally heavy task at the desired latency, the overall markerless pipeline leverages the massively parallel computation abilities of modern CPUs and GPUs, allowing us to optimize every stage of the computations involved.

Another critical aspect is the training stage of any ML approach which generally requires a lot of annotated input data. And while in the field of pose estimation a variety of public datasets is available, they are often limited to a single view (where we require multi-view data for our multi-camera setups), and their annotations are not necessarily precise. Rather than record our own real-life dataset, the project opts to leverage our expertise in the creation of avatars in what we call our Synthetic Factory. Based on a single avatar model equipped with a variety of blend shapes, we can create a large variety of distinct avatars by providing broad characteristics such as age, gender, ethnicity, etc. Add in a variety of body proportions, skin tones, outfits, footwear, hairstyles, and animations, and you get a virtually unlimited set of subjects you can record from any angle, with their underlying skeleton forming the ground-truth annotation. This dataset then forms the basis of any of our training efforts.

This project was performed in collaboration with Vicon, the world leader mocap provider. First results of this joint collaboration were presented at SIGGRAPH 2023 on their exhibition booth, showcasing a six person, markerless and multi-modal real-time solve, set against Dreamscape’s LBVR adventure called The Clockwork Forest. With this showcase, Vicon earned a CGW Silver Edge Award for technological innovation.

Partners

Artanim
VR requirements definition, ML-based tracking algorithms evaluation, synthetic factory development, multi-modal tracking solution implementation and fine-tuning

Vicon
Hardware development, ML-based tracking algorithms implementation, real-time solve