You can read every recipe, but to really learn how to cook, you need time in the kitchen. Wouldn't it be great if you could slip on a VR headset and have a famous chef walk you through the basics step by step? In the future, you might be able to learn how to cook a delicious five-course meal—all in VR. In fact, virtual reality could help people learn all kinds of skills.

At Daydream Labs, we tried to better understand how interactive learning might work in VR. So we set up an experiment, which aimed at teaching coffee making. We built a training prototype featuring a 3D model of an espresso machine which reacts like a real one would when you press the buttons, turn the knobs or drop the milk. We also added a detailed tutorial. Then, we tasked one group of people to learn how to pull espresso shots by doing it in VR. (At the end, we gave people a detailed report on how they’d done, including an analysis of the quality of their coffee.) For the purpose of comparison, another group learned by watching YouTube videos. Both groups were able to train for as long as they liked before trying to make a coffee in the real world; people assigned to watch the YouTube tutorial normally did so three times, and people who took the VR training normally went through it twice.

We were excited to find out that people learned faster and better in VR. Both the number of mistakes made and the time to complete an espresso were significantly lower for those trained in VR (although, in fairness, our tasting panel wasn't terribly impressed with the espressos made by either group!) It's impossible to tell from one experiment, of course, but these early results are promising. We also learned a lot of about how to design future experiments. Here's a glimpse at some of those insights.

First, milk coffee was a bad choice. The physical sensation of tamping simply can't be replicated with a haptic buzz. And no matter what warning we flashed if someone virtually touched a hot steam nozzle, they frequently got too close to it in the real world, and we needed a chaperone at the ready to grab their hand away. This suggests that VR technology isn’t quite there when it comes to learning some skills. Until gloves with much better tracking and haptics are mainstream, VR training will be limited to inputs like moving things around or pressing buttons. And if the digital analog is too far removed from the thing it's simulating, it probably won’t help all that much with actually learning the skill.

We also learned that people don’t follow instructions. We see this in all of the prototypes made in Daydream Labs, but it was especially problematic in the trainer. Instructions on controllers? People left their hands by their sides. Written on a backboard? They were too busy with what was right in front of them. Delivered as a voiceover? They rushed ahead without waiting. We even added a “hint” button, but people thought that it was cheating—and forgot about it after a step or two anyways. We ended up needing to combine all of these methods and add in-scene markers, too. Large green arrows pointing at whatever the user was supposed to interact with next worked well enough to allow us to run the test. But we’ve by no means solved this problem, and we learned that lots more work needs to be done about incorporating instructions effectively.

Finally, we discovered that it was too difficult to track all the steps a person took. Every choice we gave a user led to an exponential growth in the number of paths through the tutorial. Worse, people didn't always follow our linear “railroad-style” path, so we had to model all kinds of situations; for example, letting the user steam the milk before grinding the coffee. In the end, it was much easier to model the trainer like a video game, where every object has its own state. So instead of the trainer keeping track of all the steps the user did in order ("user has added milk to cup", we had it track whether a key step had been achieved ("cup contains milk").

Despite these challenges, we consider this prototype a success: people learned something new in VR and enjoyed the process. Some of them even came back to use the espresso trainer again after they’d tried to make a real coffee. In fact, once they had the real-world experience, the virtual training had more context and was more meaningful to them. It may be that VR is a useful way to introduce people to a new skill, and then can help them practice and consolidate once they’ve tried it in the real world. One thing’s for sure—there’s a lot more to learn about learning in VR!