These researchers came up with a solution for one of VR's biggest issues: tracking your legs
How virtual reality is gamifying fitnessReplayMore Videos ... (16 Videos)How virtual reality is gamifying fitnessWhy privacy experts are warning against using period-tracking appsBig Tech and Ireland: How the combination made Ireland one of Europe's wealthiest countries This mask makes breathing in virtual reality more realisticSee how Google's new AR technology worksIn 1997, an IBM computer beat a chess world champion for the first timeIn 2005, an iPod was sold every two seconds. See how CNN covered the phenomenonHear Elon Musk's plans for Trump's Twitter banKlarna CEO on why 'buy now, pay later' works Facebook's parent company has a brick and mortar store. See what's insideBill Gates: Elon Musk 'could make Twitter worse'Elon Musk set to purchase Twitter for $44 billionWatch SpaceX and Axiom land first civilian crew back on EarthA quarter of all the electricity in this county is powering Bitcoin miningLamborghini's last Aventador Coupé with an NFT sold for $1.6MMother describes son's social media addiction that 'led to his death' (CNN)If you're a newcomer to virtual reality, you'll notice something odd about your appearance: Depending on the app, you may look down to see you're represented by an avatar that's just a floating torso or nothing but a pair of hands.
While wireless headsets and handheld controllers laden with sensors and software do a good job of tracking your upper body in VR, actual leg tracking is still rare. That's because it tends to require an array of sensors on or around you to do it well, which most consumers don't want to buy. (Some apps, such as VRChat, do offer full-body avatars, but they tend to use software to approximate lower-body motions.)Now, a group of researchers at Carnegie Mellon University (CMU) are offering a possible solution to the leg tracking problem that has so far arguably limited how immersed people can really feel in virtual spaces.Researchers added cameras to Quest 2 controllers to enable full-body tracking in virtual reality, such as with this demo of Feet Saber (a reference to the game Beat Saber).The researchers came up with a way to track a person's entire body by affixing cameras with fisheye lenses to a pair of Facebook-parent Meta's Quest 2 controllers, which is currently the most popular VR headset by far (and, like most headsets on the market, does not currently offer leg tracking). As a Quest 2 headset wearer held the controllers, the cameras captured images of their body and were used to control a full-body avatar in several VR demos the researchers created.With the prototype, users could make their avatar kick and stomp blocks in "Feet Saber" (a tongue-in-cheek reference to the popular VR game "Beat Saber"), or twist and turn into the right position to pass through an obstacle in a sort of human-sized "Tetris" game. The researchers also came up with demos that let users kick a soccer ball and defend a goal from incoming hockey pucks.Read More"I think the legs are expressive. I think they are important to see how people are moving around — not just for locomotion but body pose for human-to-human interactions," Chris Harrison, an associate professor of human computer interaction at CMU and a coauthor of the study, told CNN Business.Why you can't have legs in virtual reality (yet)The prototype, called ControllerPose, isn't precise enough to capture fine movements, Harrison said, but it works "quite well" for tracking coarser actions. "If you want to look down and see if you're squatting or lifting a leg or kicking a ball, it's more than accurate enough to capture those kind of cases," he said.On average, the system is off by 6.98 centimeters (2.75 inches) when estimating where to show a joint in VR versus where it actually is, he said.To make it work, researchers streamed images from two fisheye-lens cameras on each controller — one facing upwards, one facing downwards — to a computer. (Initially, they used a desktop computer; later on, they used a small computer connected to a controller to show the whole process could be done right in the palm of your hand.) Software filtered out bad pictures and fixed lens distortion, then stitched camera images together so they could be used to estimate the headset wearer's real-life poses. Those estimates determined the poses of the full-body avatar in VR.Harrison's lab at CMU, the Future Interfaces Group, has worked on paid collaborations with Meta in the past, he said, but this was not one of those projects.Meta has been considering for years how to make avatars more realistic. In an Instagram "Ask Me Anything" session in February, Meta CTO Andrew Bosworth acknowledged the difficulty of the task while saying the company is considering how to solve it. "Tracking your own legs accurately is super hard and basically not workable just from a physics standpoint with existing headsets," Bosworth said.Researchers affixed two fisheye-lens cameras to Quest 2 controllers to enable full-body tracking in VR.Skip Rizzo, director of the Medical Virtual Reality Group at the University of California's Institute for Creative Technologies, thinks leg tracking is useful for certain applications, such as when using VR for rehabilitation after a stroke. (One of the researchers' demos was a balance-training exercise that had users stand on one leg at a time.) As with video game accessories — be it a steering wheel for a racing game or a vest that vibrates if you're shot in a first-person-shooter game — he knows that most people don't want to pay for extra sensors to use with their VR headsets. "You gotta be real hardcore to buy that," he said.Still, he noted that adding cameras and a small computer to hand controllers would suck up more power (and since the Quest 2 controllers run on batteries, that could mean replacing them more often). Harrison, too, sees power consumption as a challenge; in the future, the researchers would like to increase the speed at which they can process images on the handheld controller, but they don't want to run down its battery life too quickly. While the ControllerPose project is meant to get people interested in the potential for using controllers to track the body, Harrison thinks it's likely we'll see such a feature included with an upcoming VR headset. (Meta declined to comment.)"It's not that it's the killer app for VR," Harrison said, "but we're just going that one step closer to unlocking that vision of really teleporting yourself into another reality."Click Here To Get Funded!