Author Archives: Alesha Unpingco

Best practices for mobile AR design

Over the past few years, many people have experienced virtual reality with headsets like Cardboard, Daydream View, and higher-end PC units like Oculus Rift and HTC Vive. Now, augmented reality has the potential to reach people right on their mobile devices. AR can bring information to you, and that digital information can enhance the experience you have with their physical space. However, AR is new, so creators need to think carefully when it comes to designing intuitive user interactions.

From our own explorations, we’ve learned a few things about design patterns that may be useful for creators as they consider mobile AR platforms. For this post, we revisited our learnings from designing for head-mounted displays, mobile virtual reality experiences, and depth-sensing augmented reality applications. First-party apps such as Google Earth VR and Tilt Brush allow users to explore and create with two positionally-tracked controllers. Daydream helped us understand the opportunities and constraints for designing immersive experiences for mobile. Mobile AR introduces a new set of interaction challenges. Our explorations show how we’ve attempted to adapt emerging patterns to address different physical environments and the need to hold the phone throughout an entire application session.

Key design considerations

mobar1

Mobile constraints. Achieving immersive interactions is possible through a combination of the device's camera, real-world coordinates for digital objects, and input methods of screen-touch and proximity. Since mobile AR experiences typically require at least one hand to hold the phone at all times, it's important for interactions to be discoverable, intuitive, and easy to achieve with one or no hands. The mobile device is the user’s window into the augmented world, so creators must also consider ways to make their mobile AR experiences enjoyable and usable for varying screen sizes and orientations.

mobar2

Mobile mental models and dimension-shifts. Content creators should keep in mind existing mental models of mobile AR users. 2D UI patterns, when locked to the user’s mobile screen, tend to lead to a more sedentary application experience; however, developers and designers can get creative with world-locked UI or other interaction patterns that encourage movement throughout the physical space in order to guide users toward a deeper and richer experience. The latter approach tends to be a more natural way to get users to learn and adapt to the 3D nature of their application session and more quickly begin to appreciate the value a mobile AR experience has to offer — such as observing augmented objects from many different angles.

mobar3

Environmental considerations. Each application has a dedicated "experience space," which is a combination of the physical space and range of motion the experience requires. Combined with ARCore's ability to detect varying plane sizes or overlapping planes at different elevations, this opens the door to unique volumetric responsive design opportunities that allow creators to determine how digital objects should react or scale to the constraints of the user's mobile play space. Visual cues like instructional text or character animations can direct users to move around their physical spaces in order to reinforce the context switch to AR and encourage proper environment scanning.

mobar4

Visual affordances. Advanced screen display and lighting technology makes it possible for digitally rendered objects to appear naturally in the user’s environment. Volumetric UI patterns  can complement a 3D mobile AR experience, but it’s still important that they stand out as interactive components so users get a sense of selection state and functionality. In addition to helping users interact with virtual objects in their environment, it’s important to communicate the planes that the mobile device detects in order to manage the users’ expectations for where digital items can be placed.

mobar5

Mobile AR 2D interactions. With mobile AR, we’ve seen applications of a 2D screen-locked UI which gives users a “magic-hand” pattern to engage with the virtual world via touch inputs. The ability to interact with objects from a distance can be very empowering for users. However, because of 2D UI patterns' previous association with movement-agnostic experiences, users are less likely to move around. If physical movement is a desired form of interaction, mobile AR creators can consider ways to more immediately use plane detection, digital object depth, and phone-position to motivate exploration of a volumetric space. But be wary of too much 2D UI, as it can break immersion and disconnect the user from the AR experience.

mobar6

Mobile AR immersive interactions. To achieve immersion, we focused on core mobile AR interaction mechanics ranging from object interaction, browsing, information display, and visual guidance. It's possible to optimize for readability, usability, and scale by considering ways to use a fixed position or dynamic scaling for digital objects. Using a reticle or raycast from the device is one way to understand intent and focus, and designers and developers may find it appropriate to have digital elements scale or react based on where the camera is pointing. Having characters react with an awareness to how close the user is, or revealing more information about an object as a user approaches, are a couple great examples of how creators can use proximity cues to reward exploration and encourage interaction via movement.

What’s next?

These are some early considerations for designers. Our team will be publishing guidelines for mobile AR design soon. There are so many unique problems that mobile AR can solve and so many delightful experiences it can unlock. We’re looking forward to seeing what users find compelling and sharing what we learn along the way, too. In the meantime, continue making and breaking things!

Images in this post by Chris Chamberlain

Daydream Elements: Foundational VR Design

Daydream Elements launched at Google I/O this year. It’s a set of best practices and reusable code for some of the most fundamental things you do in VR — like walking around and interacting with the environment. So whether you're a seasoned app developer, an immersive design enthusiast, or just curious about VR, you can take  a look at the first set of interactions.

Getting around


Teleportation

Teleportation is popular in VR because it allows you to easily explore a large virtual environment. In Elements, the Daydream controller's ability to detect touch is used to make teleportation more discoverable. When you touch the touchpad, an arc is drawn to the teleportation target, so you can click to teleport to that destination point.

TeleportGif
A simple touch on the touchpad draws the teleportation arc to a destination point.

The Teleportation Element also addresses a previous downside: becoming disoriented right after you teleport. With an instant jump or a fade to black, you need a few seconds to get your bearings again. However, you can avoid this with a very fast warp effect that quickly flies you to your new location. The warp effect also helps you remain situated, and it’s fast enough to prevent any potential discomfort.

Tunneling

Tunnel1
The field of view gets smaller and the background is replaced with a grid when walking.

Motion causes some people to feel uncomfortable in VR, even if they don’t feel that way when they play video games on a TV. That’s because the living room (with its chairs, lamps, and mid-century modern asymmetrical coffee table with hairpin legs) helps keep you grounded. The Tunneling Element for VR takes advantage of this same living room effect, and can be used during rotation or when virtually moving around. In the Tunneling Element, the use of a stable grid was the most effective at reducing discomfort. This interaction model was also used and tested in Earth VR and is now available for use in any app.

Teleport2
Bringing in tunneling during rotation helps improve comfort for users who are highly sensitive to major movement.

Chase Camera

Sometimes you need to follow a character while in-game, but if it happens automatically and isn’t under your control, it may be uncomfortable. Our Chase Camera Element is fine tuned to avoid these issues. It emphasizes control and expectations: you determine your target destination with a touchpad click so that moving in that direction feels natural and expected.

CC1
The user is in control of the camera movement by selecting the target destination for the fox.

It also avoids automatic camera rotation. You can still manually rotate the camera with the touchpad, and the Tunneling Element will turn on during the rotation or if you look in a different direction from the camera movement—which is one of the biggest reasons that chase cameras are so hard to get right in VR.  Also note that tunneling effects the environment, but not the character—the fox can run into the tunnel so the user never loses it.

CC2

Menu Systems

Click Menu

The Daydream controller is simple and easy to use, but sometimes you need more action choices. Your touchpad-touches may be used for moving around, and your touchpad-click may be dedicated to a primary app action, but you still may want easy access to more options.

CM1
Clicking the app button reveals a radial menu of options that are hidden from the user at first but are always easily accessible.

In Elements, there’s an example of a drawing application where you click the app button to see a radial menu for access to a larger set of drawing tools and colors. That way, the touchpad can remain dedicated to a primary action (in this case, point-and-click drawing).

Swipe Menu

What if you're fighting a menacing fire-breathing dragon in the boss level of your favorite RPG, and you need to quickly switch between your sword and crossbow, not to mention replenish your health with your limited supply of potions? The swipe menu is a great way to quickly swap between tools. In Daydream Elements, no dragons were available, but you can see the swipe menu in action against some balloons (still menacing).

SwipeMenu
The swipe menu provides very fast access to actions.

Daydream Renderer

Last, there’s a new real time rendering system for Daydream, and it's pretty awesome. You can play with dynamic light sources and textures, and it shows that mobile devices are now capable of rich visual experiences. This demo scene is just one example of how content creators can use the Daydream Renderer to make their apps look beautiful. If you’re interested in the technical details of how all this works, check out the documentation.

DDR

The goal of Daydream Elements is to make VR design and development accessible to everyone who wants to create great immersive content. Keep your eyes peeled for more Elements coming soon!