The sensors located in the Meta Quest controllers and headset provide full 6DoF tracking for the head and hands, which allows their movement to be accurately reproduced in VR. Other parts of your body however, like the arms and legs, are not tracked in the same way. This means the movement of virtual avatars can look a bit strange, for example, the legs might drag on the floor or the arms bend in all kinds of strange and unnatural ways.  

The solution to this was to use additional sensors or external cameras to enable full-body tracking, but this may soon not be required. Meta Reality Labs have found a way to use a type of Artificial Intelligence (AI) combined with the headset sensors to predict the position of the body in various poses. This technology gives a more accurate full-body representation of a person in VR without the need for additional technology.

How it works

The Meta team used a type of AI called Machine Learning (ML), which is based on the idea that systems can identify patterns from data and make decisions based on those patterns. The team input eight hours of motion-capture data from 172 people into a ML program called QuestSim. The end result was a program with the ability to accurately predict the position of the whole body by only using the sensors located in the headset and controllers.  

In other words, realistic full-body tracking was achieved without the need for anything but the Quest. This predictive full-body tracking technology allows for better embodiment of a virtual avatar and you can see it in action for yourself by watching the video below.

Whilst this is impressive, do bear in mind that because this kind of tracking is predictive, it isn’t 100% accurate. 

INSERT VIDEO: [https://youtu.be/CkTHsz6Ldas]

More work to be done

This AI that is able to predict your body position is basing its predictions on the movement of the motion-capture clips provided, which included motion such as walking, jogging and casual chatting. This means if you move in a way that’s quite different to anything it has seen before, let’s say by doing the splits, it could have a hard time predicting what the body would look like in that pose.

Moving too fast can also cause the software to fail in accurately predicting your pose and your avatar needs to closely match your own proportions for it to work properly. Another limitation is that it doesn’t support the popular VR locomotion method of teleportation. Despite these drawbacks, the end result is promising and it will be interesting to see how this technology develops.

Future developments

Meta plans to include more skeletal and body shape data to improve the diversity of movements for more accurate body tracking predictions. It also might not be long before consumers get a chance to try this predictive full-body tracking for themselves as we are about to see the release of the Meta Quest Pro over the next month. 

The Meta Quest Pro supports eye and face tracking, so the release of this full-body tracking tech alongside it would be the perfect addition in creating avatars with more realistic expressions and motion.

As a company dedicated to providing innovative and original content, we pride ourselves in also working with influencers that amplify this dedication. Therefore we are honored to collaborate with Tony from VR Reviews helping us to share the latest virtual reality news on our website, please click the image below to find out more.