Summary

Curate XR explores how generative AI and large language models (LLMs) can enable more natural and flexible interaction with virtual agents in extended reality. The project uses a museum guide scenario as a testbed, allowing visitors to engage with an embodied virtual character through spontaneous conversation rather than predefined questions and responses.

Interaction takes place through natural speech. The virtual guide responds in real time while taking into account both predefined information about the environment and contextual data such as the user’s location and the artwork they are currently viewing. The agent can also communicate in different languages, making the interaction more intuitive and accessible to a broad range of users.

The conversational system is integrated with an animated virtual character that navigates the environment using motion matching and pathfinding. This allows the guide to move naturally through the space, accompany the user and respond not only through speech but also through embodied behavior.

Curate XR also investigates how generative AI agents can translate spoken instructions into actions. In a virtual sculpting scenario, users can create, position, move and modify three-dimensional objects simply by describing what they want. The agent interprets these requests and updates the virtual environment accordingly, enabling iterative creation through conversation.

Although the museum setting serves as the primary research scenario, the project investigates a broader interaction framework that could be applied to training, education, rehabilitation, entertainment and other XR applications where natural communication with interactive virtual agents can improve usability and engagement.

Summary

ABC-Space investigates how attention and curiosity unfold in three-dimensional space and how they influence learning in immersive virtual reality. The project focuses in particular on educational VR experiences involving embodied social virtual agents.

The research examines how social signals generated by interactive virtual characters affect cognitive and motivational processes involved in attention, memory and learning. Through new experimental paradigms in VR, the project studies the relationships between curiosity, attention, spatial representation and memory, and how these mechanisms interact in fully immersive environments.

The results aim to contribute to VR science, cognitive science and educational psychology, while informing the design of a new generation of educational virtual agents capable of supporting attention and learning more effectively.

Within ABC-Space, Artanim develops physics-based controllers trained with deep reinforcement learning to drive virtual characters, produces the immersive VR experiments, and contributes to the design of behavioral experiments evaluating their impact on attention and learning.

Partners

Joan Llobera – Artanim
Physics-based interactive virtual characters capable of producing social signals and cues.

Patrik Vuilleumier – University of Geneva, Laboratory for Behavioral Neurology and Imaging of Cognition (LABNIC)
Cognitive and affective processes governing human attention in space.

Mireille Bétrancourt – University of Geneva, Educational Technologies (TECFA)
Models of spatial representation used to form cognitive maps and their impact on educational VR experiences.

Related Publications

Deschanet C, Kreienbühl L, Vuilleumier P, Llobera J. Adapting the Posner paradigm to study 360º attention orienting in Virtual Reality, Vision Sciences Society, May 2026.
PDF

Summary

PRESENCE investigates how advanced XR technologies can strengthen the sense of presence in shared physical-digital environments. The project addresses key perceptual dimensions including plausibility, co-presence and place illusion, with the aim of making remote interaction feel increasingly natural and convincing.

The project focuses on three main challenges. First, it explores realistic visual interaction between remote users through live volumetric capture, compression and optimization techniques for high-quality holoportation under heterogeneous computing and network conditions. Second, it investigates realistic touch between remote users and virtual objects through novel haptic systems and spatial synchronization across multiple devices. Third, it develops more realistic social interaction through avatars, autonomous agents and AI-powered virtual humans.

Within PRESENCE, Artanim focuses on smart autonomous virtual characters capable of generating physically plausible behavior while responding to social and contextual cues from human-controlled avatars and other agents.

The research investigates how virtual agents can adapt their movement and behavior to interpersonal distance, social conventions and collaborative tasks. For example, an agent can adjust its position and orientation to respect proxemic rules, or recognize when a user needs an object and respond by retrieving and offering it.

Partners

Fundació i2CAT (Spain)

Actronika (France)

University of Hamburg (Germany)

Centre for Research and Technology Hellas – CERTH (Greece)

Raytrix GmbH (Germany)

SenseGlove B.V. (Netherlands)

Go Touch VR SAS (France)

Didimo S.A. (Portugal)

Vection Italy Srl (Italy)

University of Barcelona (Spain)

Unity Technologies (Denmark)

Sound Holding B.V. (Netherlands)

Interuniversitair Micro-Electronica Centrum – imec (Belgium)

JOANNEUM RESEARCH Forschungsgesellschaft mbH (Austria)

SyncVR Medical B.V. (Netherlands)

Zaubar UG (haftungsbeschränkt) (Germany)

Artanim (Switzerland)

Related Publications

Mair S, Mostajeran F, Llobera J, Slater M, Steinicke F. Laughing Together: A Pilot Study on the Role of Virtual Agents in Emotional Contagion, Conformity, and Opinion Shaping in a Virtual Stand-Up Comedy Club, 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 1479–1489, October, 2025.
PDF · DOI

Llobera J, Li K, Nagorny P, Charbonnier C, Steinicke F. A Conversational Virtual Agent with Physics-based Interactive Behaviour, 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Daejeon, South Korea, 973–974, October, 2025.
PDF · DOI

Mostajeran F, Li K, Rings S, Kruse L, Wolf E, Schmidt S, Arz M, Llobera J, Nagorny P, Charbonnier C, Fassold H, Alvarez X, Tavares A, Santos N, Orvalho J, Fernández S, Steinicke F. A toolkit for creating intelligent virtual humans in extended reality, IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Saint-Malo, France, 736–741, March, 2025.
PDF · DOI

Summary

Virtual reality opens the possibility of creating narrative experiences that combine the richness of film or theater with direct interaction between users and autonomous virtual characters. Rather than following a fully predetermined sequence, participants can observe a story unfold around them, interact with its characters and potentially influence parts of the narrative.

A key challenge is that conventional character animation techniques, largely developed for video games, do not always support the subtle and multimodal interactions that users naturally expect in immersive VR. Virtual characters need to respond convincingly to a user’s presence, movements and behavior while remaining consistent with the narrative.

The project explores new approaches to interactive character animation aimed at creating more engaging and believable virtual characters. It combines methods from computer graphics, machine learning and cognitive psychology to investigate how autonomous characters can move, react and interact naturally with users within immersive narrative environments.

Related Publications

Llobera J, Charbonnier C. Physics-based character animation and human motor control, Phys Life Rev, 46:190–219, 2023.
PDF · DOI

Llobera J, Jacquat V, Calabrese C, Charbonnier C. Playing the mirror game in virtual reality with an autonomous character, Sci Rep, 12:21329, 2022.
PDF · DOI

Llobera J, Charbonnier C. Physics-based character animation for Virtual Reality, Open Access Tools and Libraries for Virtual Reality, IEEE VR Workshop, 2022 Best Open Source Tool Award, March, 2022.
PDF · DOI

Llobera J, Booth J, Charbonnier C. Physics-based character animation controllers for videogame and virtual reality production, 14th ACM SIGGRAPH Conference on Motion, Interaction and Games, November, 2021.
PDF

Llobera J, Booth J, Charbonnier C. New Techniques in Interactive Character Animation, SIGGRAPH ’21: short course, ACM, New York, NY, USA, 16:1–6, August, 2021.
PDF · DOI

Llobera J, Charbonnier C. Interactive Characters for Virtual Reality Stories, ACM International Conference on Interactive Media Experiences (IMX 2021), ACM, New York, NY, USA, 322–325, June, 2021.
PDF · DOI