Three months ago, we launched ARCore, Google’s platform for building augmented reality (AR) experiences. There are already hundreds of apps on the Google Play Store that are built on ARCore and help you see the world in a whole new way. For example, with Human Anatomy you can visualize and learn about the intricacies of the nervous system in 3D. Magic Plan lets you create a floor plan for your next remodel just by walking around the house. And Jenga AR lets you stack blocks on your dining room table with no cleanup needed after your tower collapses.

As announced today at Google I/O, we’re rolling out a major update to ARCore to help developers build more collaborative and immersive augmented reality apps.

• Shared AR experiences:Many things in life are better when you do them with other people. That’s true of AR too, which is why we’re introducing a capability called Cloud Anchors that will enable new types of collaborative AR experiences, like redecorating your home, playing games and painting a community mural—all together with your friends. You’ll be able to do this across Android and iOS.
  

• AR all around you:ARCore now features Vertical Plane Detection which means you can place AR objects on more surfaces, like textured walls. This opens up new experiences like viewing artwork above your mantlepiece before buying it. And thanks to a capability called Augmented Images, you’ll be able to bring images to life just by pointing your phone at them—like seeing what’s inside a box without opening it.  
  

• Faster AR development:With Sceneform, Java developers can now build immersive, 3D apps without having to learn complicated APIs like OpenGL. They can use it to build AR apps from scratch as well as add AR features to existing ones. And it’s highly optimized for mobile.

Developers can start building with these new capabilities today, and you can try augmented reality apps enabled by ARCore on the Google Play Store.

There’s so much information available online, but many of the questions we have are about the world right in front of us. That’s why we started working on Google Lens, to put the answers right where the questions are, and let you do more with what you see.

Last year, we introduced Lens in Google Photos and the Assistant. People are already using it to answer all kinds of questions—especially when they’re difficult to describe in a search box, like “what type of dog is that?” or “what’s that building called?”

Today at Google I/O, we announced that Lens will now be available directly in the camera app on supported devices from LGE, Motorola, Xiaomi, Sony Mobile, HMD/Nokia, Transsion, TCL, OnePlus, BQ, Asus, and of course the Google Pixel. We also announced three updates that enable Lens to answer more questions, about more things, more quickly:

First, smart text selection connects the words you see with the answers and actions you need. You can copy and paste text from the real world—like recipes, gift card codes, or Wi-Fi passwords—to your phone. Lens helps you make sense of a page of words by showing you relevant information and photos. Say you’re at a restaurant and see the name of a dish you don’t recognize—Lens will show you a picture to give you a better idea.  This requires not just recognizing shapes of letters, but also the meaning and context behind the words. This is where all our years of language understanding in Search help.

Second, sometimes your question is not, “what is that exact thing?” but instead, "what are things like it?" Now, with style match, if an outfit or home decor item catch your eye, you can open Lens and not only get info on that specific item—like reviews—but see things in a similar style that fit the look you like.

Third, Lens now works in real time. It’s able to proactively surface information instantly—and anchor it to the things you see. Now you’ll be able to browse the world around you, just by pointing your camera. This is only possible with state-of-the-art machine learning, using both on-device intelligence and cloud TPUs, to identify billions of words, phrases, places, and things in a split second.

Much like voice, we see vision as a fundamental shift in computing and a multi-year journey. We’re excited about the progress we’re making with Google Lens features that will start rolling out over the next few weeks.

Back in January, we announced the Lenovo Mirage Solo, the first standalone virtual reality headset that runs Daydream. Alongside it, we unveiled the Lenovo Mirage Camera, the first camera built for VR180. Designed with VR capture and playback in mind, these devices work great separately and together. And both are available for purchase today.

More immersive

The Mirage Solo puts everything you need for mobile VR in a single device. You don't need a smartphone, PC, or any external sensors—just pick it up, put it on, and you're in VR in seconds.

The headset was designed with comfort in mind, and it has a wide field of view and an advanced display that’s optimized for VR. It also features WorldSense, a powerful new technology that enables PC-quality positional tracking on a mobile device, without the need for any additional sensors. With it, you can duck, dodge and lean, step backward, forward or side-to-side. All of this makes for a more natural and immersive experience, so you really feel like you’re there.

With over 350 games, apps and experiences in the Daydream library, there's tons to see and do. WorldSense unlocks new gameplay elements that bring the virtual world to life, and more than 70 of these titles make use of the technology, including Blade Runner: Revelations, Extreme Whiteout, Narrows, BBC Earth Live in VR, Fire Escape, Eclipse: Edge of Light, Virtual Virtual Reality, Merry Snowballs, and Rez Infinite. So whether you’re a gamer or an explorer, there’s something for everyone.

Point and shoot VR capture

Alongside the Mirage Solo, we worked with Lenovo to develop the first VR180 consumer camera, the Lenovo Mirage Camera. VR180 lets anyone capture immersive VR content with point and shoot simplicity. Photos and videos taken with the camera transport you back to the moment of capture with a 180 degree field of view and crisp, three-dimensional imagery.

There’s no better place to relive your VR180 memories than in the Lenovo Mirage Solo headset. And with support for VR180 built into Google Photos, you can easily share those moments with your friends and family—regardless of what device they have.

We can’t wait for you to try out the Lenovo Mirage Solo and Mirage Camera to dive into new immersive experiences, and to start capturing your favorite moments in VR.

Last weekend, fans from all around the world made the trek to Southern California to see some of music’s biggest names perform at Coachella. To make those not at the festival feel like they were there, we headed to the desert with VR180 cameras to capture all the action.

Throughout the first weekend of Coachella, we embarked on one of the largest VR live streams to date, streaming more than 25 performances (with as many cameras to boot) across 20 hours and capturing behind-the-scenes footage of fans and the bands they love. If you missed it live, you can enjoy some of the best experiences—posted here.

VR180 can take you places you never thought possible—the front row at a concert, a faraway travel destination, the finals of your favorite sporting event, or a memorable location. This year at Coachella, we pushed the format even further by adding augmented reality, AR, overlays on top of the performances—like digital confetti that falls when the beat drops, or virtual objects that extend the into the crowd.

To add these overlays in real time, we used our VR180 cameras together with stitching servers running a custom 3D graphics engine and several positionally tracked cameras. This allowed us to add a layer of spatially relevant visuals to the video feed. Simply put, it's like AR stickers for VR180.

In addition to the responsive AR elements during performances, we also featured Tilt Brush art by artist-in-residence Cesar Ortega, who drew his live impressions of the iconic Coachella landscape at daylight, dusk and night. We then inserted Cesar’s designs into the video stream in VR180 to allow the viewer to see the art.

Watch the festival footage, including performances and behind-the-scenes footage from the point of view of both the fans and the bands, here. And for the most immersive experience, check it out in VR with Daydream View or Cardboard.

Last year we introduced VR180, a new video format that makes it possible to capture or create engaging immersive videos for your audience. Most VR180 cameras work just like point-and-shoot models. However, what you capture in VR180 is far more immersive. You’re able to create VR photos and videos in stunning 4K resolution with just the click of a button.

Today, we’re publishing the remaining details about creating VR180 videos on github and photos on the Google Developer website, so any developer or manufacturer can start engaging with VR180.

For VR180 video, we simply extended the Spherical Video Metadata V2standard. Spherical V2 supports the mesh-based projection needed to allow consumer cameras to output raw fisheye footage. We then created the Camera Motion Metadata Track so that you’re able to stabilize the video according to the camera motion after video capture. This results in a more comfortable VR experience for viewers. The photos that are generated by the cameras are written in the existing VR Photo Format pioneered by Cardboard Camera.

When you use a Cardboard or Daydream View to look back on photos and videos captured using VR180, you’ll feel like you’re stepping back into your memory. And you can share the footage with others using Google Photos or YouTube, on your phone or the web. We hope that this makes it simple for anyone to shoot VR content, and watch it too.

In the coming months, we will be publishing tools that help with writing appropriately formatted VR180 photos and videos and playing it back, so stay tuned!

We announced Jump in 2015 to simplify VR video production from capture to playback. High-quality VR cameras make capture easier, and Jump Assembler makes automated stitching quicker, more accessible and affordable for VR creators. Using sophisticated computer vision algorithms and the computing power of Google's data centers, Jump Assembler creates clean, realistic image stitching resulting in immersive 3D 360 video.

Stitching, then and now

Let’s first take a look at how our standard algorithm works. It’s based on the concept of optical flow, which matches pixels in one image to those in another. When matched, you can tell how pixels “moved” or “flowed” from one image to the next. And once every pixel is matched, you can interpolate the in-between views by shifting the pixels part of the way. This means that you can “fill in the gaps” between the cameras on the rig, so that, when stitched together, the result is a seamless, coherent 360° panorama.

Using depth for better stitches

Instead, the new, multi-view stereo algorithm computes the depthof each pixel (e.g., the distance to the object at that pixel, a 3D point), and any camera on the rig that sees that 3D point can help to establish it’s depth, making the matching process more reliable.

The new approach also helps resolve a key challenge for any stitching algorithm: occlusion. That is, handling objects that are visible in one image but not in another. Multi-view stereo stitching is better at dealing with occlusion because if an object is hidden in one image, the algorithm can use an image from any of the surrounding cameras on the rig to determine the correct depth of that point. This helps reduce stitching artifacts and produce depth maps with clean object boundaries.

If you’re a VR filmmaker and want to try this new algorithm for yourself, select “high quality” in the stitching quality dropdown in Jump Manager for your next stitch!