Vicon debuts its markerless motion capture technology in collaboration with Artanim, the research arm of Dreamscape, the leading VR company, at SIGGRAPH 2023
Vicon today marks a new era of innovation in the field of motion capture, announcing that it will debut its machine learning (ML) powered markerless technology at SIGGRAPH 2023 in Los Angeles, in collaboration with Artanim, the Swiss research institute and the award-winning VR group, Dreamscape Immersive.
The announcement follows nearly three years of research and development focusing on the integration of ML and artificial intelligence (AI) into markerless motion capture at Vicon’s renowned R&D facility in Oxford, UK. The project has been undertaken by a new dedicated team, led by CTO Mark Finch, and leverages nearly four decades worth of Vicon innovation to enable a seamless, end-to-end markerless infrastructure, from pixel to purpose.
Commenting on the news, Imogen Moorhouse, Vicon CEO, said:
“Today marks the beginning of a new era for motion capture. The ability to capture motion without markers, while maintaining industry leading accuracy and precision, is an incredibly complex feat. After an initial research phase, we have focused on developing the world class markerless capture algorithms, robust real time tracking, labeling and solving needed to make this innovation a reality. What we are demonstrating at SIGGRAPH is not a one-off concept, or simply a technology demonstrator. It is our first step towards future product launches which will culminate in a first-of-its-kind platform for markerless motion capture.”
For Dreamscape, markerless motion capture can now provide a more true-to-life adventure than any other immersive VR experience by allowing for more free-flowing movement and exploration with even less user gear. Commenting on the decision to partner with Vicon, Aaron Grosky, President & COO of Dreamscape said:
“We have been anxiously awaiting the time when markerless could break from concept and into product, where the technology could support the precision required to realize its amazing potential. Vicon’s reputation for delivering the highest standards of motion capture technology for nearly forty years and Dreamscape’s persistent quest to bring the audience into the experience with full agency and no friction meant that working together on this was a no-brainer. We’re thrilled with the result. The implications for both quality of experience and ease of operations across all our efforts, from location-based entertainment to transforming the educational journey with Dreamscape Learn, is just game-changing.”
The collaboration between Vicon and Artanim was key to ensure the desired requirements for the VR use case were met.
“Achieving best in class virtual body ownership and immersion in VR requires both accurate tracking and very low latency. We spent substantial R&D effort evaluating computational performance of ML-based tracking algorithms, implementing and fine-tuning the multi-modal tracking solution, as well as taking the best from the full-body markerless motion capture, and VR headset tracking capabilities,” said Sylvain Chagué, co-founder and CTO of Artanim and Dreamscape.
At SIGGRAPH, Vicon and Artanim showcased a six person, markerless and multi-modal real-time solve, set against Dreamscape’s location-based virtual reality adventure called The Clockwork Forest, created in partnership with Audemars Piguet, the Swiss haute horology manufacturer. The experience allows participants to shrink to the size of an ant and explore a mechanical wonderland, while working together to repair the Source of Time and restore the rhythm of nature.
With this showcase, Vicon earned a CGW Silver Edge Award for technological innovation.More information about the collaborative R&D project here.
Dreamscape Immersive, the virtual reality company backed by major Hollywood studios, announces the opening of its first European venue in Geneva in July 2022. Halfway between cinema and theme-park, Dreamscape offers its visitors a virtual reality experience like no other that combines the power of immersive storytelling, motion capture and virtual reality (VR) technologies.
The opening of the Dreamscape center in Geneva, in Confederation Center, is planned in two stages. The 1st of July will see the return of the Geneva 1850 VR experience. Created by Artanim and originally premiered at the Museum of Art and History in 2019 (read also here), this widely acclaimed experience has been enhanced to include new characters, new sound design, and two minutes of new content. Then, in September 2022, the center will offer two additional adventures designed by Dreamscape Immersive: Alien Zoo, as well as DreamWorks Dragons Flight Academy – which promise the thrills and magic of immersive and full-sensory storytelling.
The Geneva 1850 experience immerses visitors in the heart of the 19th century, at the time of the Fazy revolution. Visitors, welcomed by General Guillaume Henri Dufour, will have a front row seat to the upheavals that shook the City of Calvin. The story leads them from the top of the Salève mountain to the shores of Lake Geneva, passing through the alleys of the Old Town, in the Geneva of the famous Magnin relief of 1850 which served as the basis for the setting of the adventure.
Alien Zoo transports visitors to an orbiting zoo populated by all kinds of extraordinary and wonderful creatures, some of them dangerous, but all of them on the verge of extinction. Upon arrival, travelers can play with exotic and majestic creatures and join forces to narrowly escape the galaxy’s greatest predator.
DreamWorks Dragons Flight Academy has staged a rescue mission in the Dragons license universe. Aboard their flying mount, each participant will ride a dragon and soar in free flight towards the Hidden World to save the village from invaders, while trying not to set fire to their virtual world.
“Dreamscape was founded to be a place where people make lifelong memories by exploring fantastical worlds with their friends and family,” said Walter Parkes, co-founder and CEO of Dreamscape. “The genesis of Dreamscape was born in Geneva, so it’s particularly meaningful to be opening this location in the place where it all began.”
A virtual reality experience like no other
A true meeting place in the heart of Geneva, Dreamscape offers a new type of high-end entertainment. Equipped with VR equipment and motion sensors, participants are immersed in universes that push the limits of virtual reality. During the experience, they can interact with other participants, feel the wind on their skin, grab an object, move freely or even choose the appearance of their avatar.
The programming of Dreamscape in Geneva is evolving and new adventures will be offered regularly.
Caecilia Charbonnier, co-founder of Dreamscape and Artanim specifies that “after the opening of four centers in the United States and one in Dubai, we are opening the first European center in Geneva in order to get closer to our technological development, where the adventure began. This center will be our laboratory. The Geneva public will be able to test new adventures and technological innovations for the first time.”
Geneva, laboratory of innovation
The Dreamscape center in Geneva was out of a partnership between Dreamscape Immersive, ACE & Company, Opus One and Artanim. With the opening of this center, the partners aim to promote the technological and creative ecosystem of the region while offering synergies with universities and companies in the Lake Geneva region at the forefront of digital transformation.
“Dreamscape invents new worlds and new ways of telling stories. Technology is put to work for the emotions by pushing the limits of the genre and blurring the boundaries between reality and imagination. The result is truly amazing,” explains Vincent Sager, director of Opus One. “It is this approach that convinced us to embark on the adventure and to accompany Dreamscape in the opening of their first European center, here in Geneva.”
“I am personally very happy to be able to support this project and ensure that the first Dreamscape in Europe opens its doors in Geneva, which seems relevant to me for a company whose world-famous technology was invented on our shores by our local entrepreneurs.” adds Adam Said, co-founder and CEO of ACE & Company.
Invitation to the press
Press representatives are invited, on Monday July 4, 2022 at 10 a.m., to discover the Dreamscape center in Geneva in exclusivity. Please reserve your place until June 24, 2022 by registering by email at media@dreamscapegeneva.com. Download the media kit here.
Virtual avatars play a key role in many of our activities. They embody the actors in our productions, serve as reliable subjects in our machine learning efforts, or are the virtual bodies we inhabit when going on an adventure in one of our experiences. In cases where an avatar needs to represent a specific character or historical figure, the skilled work of our artists is often sufficient to create the final result.
Sometimes however we need a lot of variation. Whether this is to allow users to create avatars which more closely match their own appearance, or when creating datasets with an appropriate – and often overlooked – diversity for machine learning efforts. In such cases, the creation of tailor-made avatars is no longer feasible. It is with this in mind that we started what we call our Synthetic Factory efforts.
The basic premise of the Synthetic Factory is a simple idea. Given a neutral default character, and a set of outfits, footwear, and hair styles it can wear, can we create a set of size, shape, gender, and ethnicity changes which can be applied in real-time to create any subject we would like? The potential benefit is clear to see. Artists can simply design an outfit or a hairstyle which applies well onto a base model, and the Synthetic Factory would take care of the variations.
At the basis of the Synthetic Factory is a default avatar model. Artists can take this model and deform it to give it a certain characteristic. Whether this is a gender appropriate change in body type, a specific set of facial features, or an overall change in body size. The synthetic factory takes these inputs, as well as meta-data characterizing the “deformation”, and encodes them as blend shapes which can be applied on a weighted bases, smoothly blending between the original shape and the blend shape target. At runtime either a specific set of deformations and weights can be selected, or a set of broad characteristics can be supplied – such as “Female”, “Adult”, “Indian” for example – after which random deformations and weights are selected which are appropriate for those characteristics.
The base model (center) with 2 of its 80 blendshape.
Any assets applied on top of this model, be they outfits, footwear, hair styles and the like, need to be able to deform appropriately given any changes made to the body blend shapes. The Synthetic Factory automates this process in a pre-processing step. After applying the asset to the neutral base model, a mapping is created, encoding how the asset fits the model. Once this mapping is complete, the factory runs through all the body blend shapes, and then determines how the asset needs to deform to keep the appropriate mapping in relation to the body.
Three avatars wearing the same outfit at 3 different proportions.
These deformations are then once again stored as blend shapes, this time as a part of the asset. This precomputation happens once, and at runtime no further heavy computation is required. All assets get annotated with a set of meta-data, making sure they are only ever combined in a way matching the model’s chosen characteristics.
As a part of its run-time tools, the Synthetic Factory also enables the modification of the skeleton driving the avatar. Either through specification of exact body measurements which match a specific dataset, or by supplying an overall target height, what we call the “AvatarBuilder” will generate a new skeleton matching the requirements and applies it to the Avatar.
Combine these tools with more fine-grained randomizations such as appropriate skin tones, as well as color and texture randomizations on outfits, and from a relatively compact set of basic inputs you have the ability to generate a large variety of diverse avatars fit for any purpose. In the video below, you can see the result for 36 randomly created avatars based on gender, age and ethnicity input.
In November 1620, the passengers and crew of the Mayflower reached the coast of Massachusetts and established a colony there. Among them were Puritan Reformists who had crossed the ocean to found a community in accordance with their aspirations, firmly rooted in Protestant and Calvinist values. This was a founding moment for what was to become the United States of America.
Experience first-hand this poignant event and the identity to which it gave rise through a multifaceted exhibition at the International Museum of the Reformation in Geneva.
Aboard the Mayflower is one piece of this exhibition. For 5 minutes, embark in the company of the first Reformed community in America on the emblematic boat that crossed the Atlantic in 1620 – an unforgettable virtual reality experience created by Artanim.
More info
October 28th, 2020 – February 28th, 2021
International Museum of the Reformation (MIR), Rue du Cloître 4, 1204 Geneva.
Opening hours: Tuesday – Sunday from 10am to 5pm.
Free admission for visitors with an admission ticket for the permanent collection.
At the heart of the VR-Together project lies the objective to enable social VR experiences with strong feelings of immersion as well as co-presence. To achieve this strong social sense of sharing a space together, photorealistic real-time representations of users are used, rather than relying on abstract avatars as found in such offerings as Facebook Spaces or AltspaceVR. Using state-of-the-art technologies developed by consortium partners and off-the-shelf hardware such as Microsoft Kinect or Intel RealSense sensors, users are scanned in real-time and the captured representations are processed and streamed as point clouds or time varying meshes (TVM). These approaches to user representation, combined with HMD removal technology, allow users sharing the virtual space – while in geographically separate locations – to see each other in all three dimensions.
Early feedback from users of the Pilot 1 demonstrations regarding the ability to see themselves and others, has been positive. The question still remains however, whether or not accurate self-representation has a significant positive impact on your sense of immersion, co-presence and the overall quality of experience. Both when seeing yourself as well as when interacting with others sharing the same virtual environment with you.
To answer this question, VR-Together consortium partner Artanim will this summer run an experiment in which users will be placed in a virtual environment in which they are virtualized by a representation of themselves at varying levels of realism and likeness.
User representations will be created at 3 different levels of accuracy:
An abstract avatar-like representation which does not match the participant
A realistic representation of the participant
An in-between more abstract – perhaps cartoon-like – representation of the participant, which is still recognizable, but steers clear of such undesirable effects as the “Uncanny Valley”.
To evaluate self-representation, single users will be placed in a virtual environment in which, by means of a virtual mirror, they will be able to observe themselves. The question there is whether or not an increased likeness improves the overall VR experience. To evaluate the importance of avatar likeness in the representation of others, pairs of users who know each other (i.e. friends or family) will share a virtual environment together, again being represented at varying levels of likeness. The goal there is to understand the effects on such aspects as immersion, togetherness and quality of interaction.
The proposed experiment will help us better understand what scenarios benefit most from realistic and recognizable user representation in Virtual Reality experiences, and to what extent realism is desirable in social VR.
We are proud to announce the opening of the VR experience Geneva 1850: A Revolutionary Journey co-produced by Artanim and the Museums of Art and History of Geneva. The immersive installation will be open to public starting from April 12th at Maison Tavel, the oldest standing building in the city of Calvin.
Headset strapped to your head, movement sensors attached to your wrists and feet, you are now all geared up to dive into the Geneva 1850 experience. This VR installation immerses users into the Geneva of a time when the city was on the brink of modernity. In groups of four, impersonating period costume-clad avatars, users are given a chance to discover the city as it was in October 1846, just days before its people rose up in insurrection. Featuring a spectacular number of special effects, the experience is not only a visual, auditory and olfactory one. It also includes a physical and haptic dimension — users can feel themselves walking around the virtual city and pick up actual objects to interact with their environment, their fellow travellers and city dwellers.
Based on 3D data collected through the digitization of the Magnin model exhibited at Maison Tavel, this extraordinary experience is an invitation to travel back in time and witness the outbreak of Geneva’s “Fazy” revolution.
More info
April 12th-July 14th, 2019 – Maison Tavel, Rue du Puits-Saint-Pierre 6, 1204 Geneva.
August 1st-September 29th, 2019 – Museum of Art and History, Rue Charles-Galland 2, 1206 Geneva.
Open from 11am to 6pm, closed on Monday.
Reserve your tickets for a journey through time at MAH-GENEVE.CH
Approximate duration: 45 min.
2/ Presentation 2: "SIGGRAPH 2023 HIGHLIGHTS X DREAMSCAPE” Witness the magic of real-time markerless motion capture in this jaw-dropping reel. See what the people who went through the experience had to say. Thanks to @visitdreamscape and @artanim_mocap🔥 #Siggraph2023
We re excited to announce we have been given a Silver Edge Award award for innovation in the computer graphics industry for our markerless demonstration at @siggraph. Thanks @CGWmagazine and our friends at @artanim_mocap and @VisitDreamscape #beyondmotion #viconpowered
Congratulations to @Vicon and everybody @artanim_mocap and @VisitDreamscape who contributed to this! #bestinshow
Vicon’s Markerless Motion Capture System receives CGW Silver Edge Award at #SIGGRAPH2023 https://www.cgw.com/Press-Center/Web-Exclusives/2023/Vicon-s-Markerless-Motion-Capture-System-receive.aspx
