Build transformative augmented reality experiences with new ARCore and geospatial features

  Posted by Eric Lai, Group Product Manager

With ARCore, Google’s platform for building augmented reality experiences, we continue to enhance the ways we interact with information and experience the people and things around us. ARCore is now available on 1.4 billion Android devices and select features are also available on compatible iOS devices, making it the largest cross-device augmented reality platform.

Last year, we launched the ARCore Geospatial API, which leverages our understanding of the world through Google Maps and helps developers build AR experiences that are more immersive, richer, and more useful. We further engaged with all of you through global hackathons, such as the ARCore Geospatial API Challenge, where we saw a number of high quality submissions across a number of use cases, including gaming, local discovery, and navigation.

Today, we are introducing new ARCore Geospatial capabilities, including Streetscape Geometry API, Geospatial Depth API, and Scene Semantics API to help you build transformative, world-scale immersive experiences.


Introducing Streetscape Geometry API

With the new Streetscape Geometry API, you can interact, visualize, and transform building geometry around the user. The Streetscape Geometry API makes it easy for developers to build experiences that interact with real world geometry, like reskinning buildings, power more accurate occlusion, or just placing a virtual asset on a building, by providing a 3D mesh within a 100m radius of the user’s mobile device location.

moving image showing streetscape geometry
Streetscape Geometry API provides a 3D mesh of nearby buildings and terrain geometry

You can use this API to build immersive experiences like transforming building geometry into live plants growing on top of them or using the building geometry as a feature in your game by having virtual balls bounce off and interact with them.

Streetscape Geometry API is available on Android and iOS.


Introducing Rooftop Anchors and Geospatial Depth

Previously, we launched Geospatial anchors which allow developers to place stable geometry at exact locations using latitude, longitude, and altitude. Over the past year, we added Terrain anchors which are placed on Earth's terrain, using only longitude and latitude coordinates, with the altitude being calculated automatically.

Today we are introducing a new type of anchor: Rooftop anchors. Rooftop anchors let you anchor digital content securely to building rooftops, respecting the building geometry and the height of buildings.

moving image showing rooftop anchors
Rooftop anchors make it easier to
anchor digital content to building rooftops
moving image showing geospatial depth
Geospatial depth combines
real time depth measurement from
users' device with Streetscape Geometry data
to generate a depth map of up to 65 meters

In addition to new anchoring features, we are also leveraging the Streetscape Geometry API to improve one of the most important capabilities in AR: Depth. Depth is critical to enable more realistic occlusion or collision of virtual objects in the real world.

Today, we are launching Geospatial Depth. It combines the mobile device real time depth measurement with Streetscape Geometry data to improve depth measurements using building and terrain data providing depth for up to 65m. With Geospatial Depth you can build increasingly realistic geospatial experiences in the real world.

Rooftop Anchors are available on Android and iOS. Geospatial Depth is available on Android.


Introducing Scene Semantics API

The Scene Semantics API uses AI to provide a class label to every pixel in an outdoor scene, so you can create custom AR experiences based on the features in an area around your user. At launch, twelve class labels are available, including sky, building, tree, road, sidewalk, vehicle, person, water and more.

moving image showing streetscape geometry
Scene Semantics API uses AI to provide accurate labels for different features that are present in a scene outdoors

You can use the Scene Semantics API to enable different experiences in your app. For example, you can identify specific scene components, such as roads and sidewalks to help guide a user through the city, people and vehicles to render realistic occlusions, the sky to create a sunset at any time of the day, and buildings to modify their appearance and anchor virtual objects.

The Scene Semantics API is available on Android.


Mega Golf: The game that brings augmented mini-golf to your neighborhood

To help you get started, we’re also releasing Mega Golf, an open source demo that helps you experience the new APIs in action. In Mega Golf you will use buildings in your city to bounce off and propel a golf ball towards a hole while avoiding 3D virtual obstacles. This open source demo is available on GitHub. We're excited to see what you can do with this project.

moving image showing streetscape geometry
Mega Golf uses Streetscape Geometry API to transform neighborhoods into a playable mini golf course where players use nearby buildings to bounce and propel a golf ball towards a hole

With these new ARCore features improvements and the new Geospatial Creator in Adobe Aero and Unity, we’ll make it easier than ever for developers and creators to build realistic augmented reality experiences that delight and provide utility for users. Get started today at g.co/ARCore. We’re excited to see what you create when the world is your canvas, playground, gallery, or more!

Introducing the Watch Face Format for Wear OS

Posted by Anna Bernbaum, Product Manager

We are excited to announce the launch of the Watch Face Format! We worked in partnership with Samsung to introduce a new way for you to build watch faces for Wear OS smartwatches.

The Watch Face Format is a declarative XML format to design the appearance and behavior of watch faces. This means that there is no executable code involved in creating a watch face, and there will be no code embedded in your watch face APK.

The Wear OS platform takes care of the logic needed to render the watch face so you no longer have to worry about code optimizations or battery performance.

Watch faces that are built with this new format require less maintenance and fewer updates than the ones built using the Jetpack Watch Face library. For example, you no longer need to update your watch face to benefit from improvements in performance or battery consumption, or to get the latest bug fixes.

Starting today, you can build watch faces in this new format and publish them on Google Play, ready for when the first Wear OS 4 watches are available.

The Watch Face Format lets you create…

Analog and digital watch faces:

Three watch faces illustrating different analogue and digital styles.

Watch faces with complications:

Three watch faces displaying different complication formats.

Customizable watch faces:

Three instances of the same watch face displayed in different color choices.

And more...

A planets-based watch face, a rainbow-inspired watch face and one based on the sun and moon

Watch Face Editing

With the Watch Face Format, we have included the watch face editor as part of Wear OS itself, so users can customize every watch face using the same editor UI. You no longer need to build your own watch face editor for users to customize their watch face.

Customizing the watch face with the in-built watch face editor
Wear OS 4’s editor for watch faces made using the Watch Face Format

Build Watch Faces, or Watch Face Tools

The new Watch Face Format can be used to build watch faces directly, or it can be integrated into creation tools, allowing designers to create watch faces without having to write any executable code.


Watch Face Studio

Today, Samsung has released the latest version of Watch Face Studio, ready for you to try now. As an alternative to directly writing XML using the Watch Face Format, Watch Face Studio makes it easy for designers to create watch faces without any coding experience.

Watch faces made in the latest version of Watch Face Studio use the Watch Face Format by default when they are run on a Wear OS 4 watch, or they run as traditional watch faces when the watch face runs on a Wear OS 3 watch.

Using Watch Face Studio to create a watch face.
Using Watch Face Studio to create a watch face

Learn more

Build watch faces using the Watch Face Format today:

I/O 2023: What’s new in Android security and privacy


Android is built with multiple layers of security and privacy protections to help keep you, your devices, and your data safe. Most importantly, we are committed to transparency, so you can see your device safety status and know how your data is being used.

Android uses the best of Google’s AI and machine learning expertise to proactively protect you and help keep you out of harm’s way. We also empower you with tools that help you take control of your privacy.

I/O is a great moment to show how we bring these features and protections all together to help you stay safe from threats like phishing attacks and password theft, while remaining in charge of your personal data.

Safe Browsing: faster more intelligent protection

Android uses Safe Browsing to protect billions of users from web-based threats, like deceptive phishing sites. This happens in the Chrome default browser and also in Android WebView, when you open web content from apps.

Safe Browsing is getting a big upgrade with a new real-time API that helps ensure you’re warned about fast-emerging malicious sites. With the newest version of Safe Browsing, devices will do real-time blocklist checks for low reputation sites. Our internal analysis has found that a significant number of phishing sites only exist for less than ten minutes to try and stay ahead of block-lists. With this real-time detection, we expect we’ll be able to block an additional 25 percent of phishing attempts every month in Chrome and Android1.

Safe Browsing isn’t just getting faster at warning users. We’ve also been building in more intelligence, leveraging Google’s advances in AI. Last year, Chrome browser on Android and desktop started utilizing a new image-based phishing detection machine learning model to visually inspect fake sites that try to pass themselves off as legitimate log-in pages. By leveraging a TensorFlow Lite model, we’re able to find 3x more2 phishing sites compared to previous machine learning models and help warn you before you get tricked into signing in. This year, we're expanding the coverage of the model to detect hundreds of more phishing campaigns and leverage new ML technologies.

This is just one example of how we use our AI expertise to keep your data safe. Last year, Android used AI to protect users from 100 billion suspected spam messages and calls.3

Passkeys helps move users beyond passwords

For many, passwords are the primary protection for their online life. In reality, they are frustrating to create, remember and are easily hacked. But hackers can’t phish a password that doesn’t exist. Which is why we are excited to share another major step forward in our passwordless journey: Passkeys.

Passkeys combine the advanced security of 2-Step Verification with the convenience of simply unlocking your device — so signing in is as easy as glancing at your phone or scanning your fingerprint. And because they use cutting-edge cryptography to create a “key” that is unique between you and a specific app or website, passkeys can’t be stolen by hackers the way that passwords can.

Last week, we announced you can use a passkey to log in to your Google Account on all major platforms. We’re the first major tech company to simplify sign-in with passkeys across our own platform. You can also use passkeys on services like PayPal, Shopify, and Docusign, with many more on the way. Start saying goodbye to passwords and try it today.

To help support developers as they incorporate passkeys, we’ve launched a Credential Manager Jetpack API that brings together multiple sign-in methods, such as passkeys, passwords and federated sign in, into a unified interface for users and a single API for developers.

Better protections for apps

Accessibility services are helpful for people with disabilities but their broad powers can be used by malware and bad apps to read screen content. In Android 14, we’re introducing a new API that lets developers limit accessibility services from interacting with their apps. Now, with a new app attribute, developers can limit access to only apps that have declared and have been validated by Google Play Protect as accessibility tools. This adds more protection from side-loaded apps that may get installed and are trying to access sensitive data.

In Android 14, we’re preventing apps that target an SDK level lower than 23 from being installed. This is because malware often targets older levels to get around newer security and privacy protections. This won’t affect existing apps on your device, but new installs will have to meet this requirement.

More transparency around how your data is used

We launched the Data safety section in Google Play last year to help you see how developers collect, share, and protect user data. Every day, millions of users use the Data Safety section information to evaluate an app’s safety before installing it.

In Android 14, we’re extending this transparency to permission dialogs, starting with location data usage. So every time an app asks for permission to use location data, you’ll be able to see right away if the app shares the location data with third parties.

And if an app changes its data sharing practices, for example, to start using it for ads purposes, we’ll notify you through a new monthly notification. As with the permissions dialogs, we’re starting with location data but will be expanding to other permission types in future releases.



We’re also empowering you with greater clarity and control over your account data by making it easier to delete accounts that you’ve created in apps. Developers will soon need to provide ways for you to ask for your account and data to be deleted via the app and the app’s Data safety section in Google Play, giving you more control both inside and outside of apps. They can also offer you an option to clean up your account and ask for other data, like activity history or images, to be deleted instead of your entire account.

Better control and protection over your photos and videos

Last year, we announced the Android Photo Picker, a new tool that apps can use to request access to specific photos and videos instead of requesting permission to a users' entire media library. We’re updating Photo Picker through Google Play services to support older devices going back to Android 4.4.

With Android 14, we modified the photo/video permissions to let you choose only specific media to share, even if an app hasn’t opted into Photo Picker. You can still decide to allow or deny all access to photos but this provides more granular control.

We’re also introducing a new API that will enable developers to recognize screenshots without requiring them to get access to your photos. This helps limit media access for developers while still providing them with the tools they need to detect screenshots in their apps.

Android remains committed to protecting users by combining advanced security and AI with thoughtful privacy controls and transparency to protect billions of users around the world. Stay tuned for more upcoming protections we’ll be launching throughout the year and learn more about how Android keeps you safe at android.com/safety.

Notes


  1. Based on estimated daily increase across desktop and mobile comparing Safe Browsing API 5 to API 4 

  2. Based on internal data from January to May 2023." 

  3. Estimating from annual and monthly spam call and spam messaging data