Category Archives: Android Developers Blog

An Open Handset Alliance Project

On-device machine learning solutions with ML Kit, now even easier to use

Posted by Christiaan Prins, Product Manager, ML Kit and Shiyu Hu, Tech Lead Manager, ML Kit

ML Kit logo

Two years ago at I/O 2018 we introduced ML Kit, making it easier for mobile developers to integrate machine learning into your apps. Today, more than 25,000 applications on Android and iOS make use of ML Kit’s features. Now, we are introducing some changes that will make it even easier to use ML Kit. In addition, we have a new feature and a set of improvements we’d like to discuss.

A new ML Kit SDK, fully focused on on-device ML

ML Kit API Overview

ML Kit's APIs are built to help you tackle common challenges in the Vision and Natural Language domains. We make it easy to recognize text, scan barcodes, track and classify objects in real-time, do translation of text, and more.

The original version of ML Kit was tightly integrated with Firebase, and we heard from many of you that you wanted more flexibility when implementing it in your apps. As a result, we are now making all the on-device APIs available in a new standalone ML Kit SDK that no longer requires a Firebase project. You can still use both ML Kit and Firebase to get the best of both products if you choose to.

With this change, ML Kit is now fully focused on on-device machine learning, giving you access to the unique benefits that on-device versus cloud ML offers:

  • It’s fast, unlocking real-time use cases- since processing happens on the device, there is no network latency. This means, we can do inference on a stream of images / video or multiple times a second on text strings.
  • Works offline - you can rely on our APIs even when the network is spotty or your app’s end-user is in an area without connectivity.
  • Privacy is retained: since all processing is performed locally, there is no need to send sensitive user data over the network to a server.

Naturally, you still get access to Google’s on-device models and processing pipelines, all accessible through easy-to-use APIs, and offered at no cost.

All ML Kit resources can now be found on our new website where we made it a lot easier to access sample apps, API reference docs and our community channels that are there to help you if you have questions.

Object detection & tracking gif Text recognition + Language ID + Translate gif

What does this mean if I already use ML Kit today?

If you are using ML Kit for Firebase’s on-device APIs in your app today, we recommend you to migrate to the new standalone ML Kit SDK to benefit from new features and updates. For more information and step-by-step instructions to update your app, please follow our Migration guide. The cloud-based APIs, model deployment and AutoML Vision Edge remain available through Firebase Machine Learning.

Shrink your app footprint with Google Play Services

Apart from making ML Kit easier to use, developers also asked if we can ship ML Kit through Google Play Services resulting in a smaller app footprint and the model can be reused between apps. Apart from Barcode scanning and Text recognition, we have now added Face detection / contour (model size: 20MB) to the list of APIs that support this functionality. 

// Face detection / Face contour model
// Delivered via Google Play Services outside your app's APK…
implementation 'com.google.android.gms:play-services-mlkit-face-detection:16.0.0'

// …or bundled with your app's APK
implementation 'com.google.mlkit:face-detection:16.0.0'

Jetpack Lifecycle / CameraX support

Android Jetpack Lifecycle support has been added to all APIs. Developers can use addObserver to automatically manage teardown of ML Kit APIs as the app goes through screen rotation or closure by the user / system. This makes CameraX integration easier. With this release, we are also recommending that developers adopt CameraX in their apps due to the ease of integration and image quality improvements (compared to Camera1) on a wide range of devices. 

// ML Kit now supports Lifecycle
val recognizer = TextRecognizer.newInstance()
lifecycle.addObserver(recognizer)

// ...

// Just like CameraX
val camera = cameraProvider.bindToLifecycle( /* lifecycleOwner= */this,
    cameraSelector, previewUseCase, analysisUseCase)

For an overview of all recent changes, check out the release notes for the new SDK.

Codelab of the day - ML Kit x CameraX

To help you get started with the new ML Kit and its support for CameraX, we have created this code lab to Recognize, Identify Language and Translate text. If you have any questions regarding this code lab, please raise them at StackOverflow and tag it with [google-mlkit]. Our team will monitor this.

screenshot of app running

Early access program

Through our early access program, developers have an opportunity to partner with the ML Kit team and get access to upcoming features. Two new APIs are now available as part of this program:

  • Entity Extraction - Detect entities in text & make them actionable. We have support for phone numbers, addresses, payment numbers, tracking numbers, date/time and more.
  • Pose Detection - Low-latency pose detection supporting 33 skeletal points, including hands and feet tracking.

If you are interested, head over to our early access page for details.

pose detection on man jumping rope

Tomorrow - Support for custom models

ML Kit's turn-key solutions are built to help you take common challenges. However, if you needed to have a more tailored solution, one that required custom models, you typically needed to build an implementation from scratch. To help, we are now providing the option to swap out the default Google models with a custom TensorFlow Lite model. We’re starting with the Image Labeling and Object Detection and Tracking APIs, that now support custom image classification models.

Tomorrow, we will dive a bit deeper into how to find or train a TensorFlow Lite model and use it either with ML Kit, or with Android Studio’s new ML binding functionality.

On-device machine learning solutions with ML Kit, now even easier to use

Posted by Christiaan Prins, Product Manager, ML Kit and Shiyu Hu, Tech Lead Manager, ML Kit

ML Kit logo

Two years ago at I/O 2018 we introduced ML Kit, making it easier for mobile developers to integrate machine learning into your apps. Today, more than 25,000 applications on Android and iOS make use of ML Kit’s features. Now, we are introducing some changes that will make it even easier to use ML Kit. In addition, we have a new feature and a set of improvements we’d like to discuss.

A new ML Kit SDK, fully focused on on-device ML

ML Kit API Overview

ML Kit's APIs are built to help you tackle common challenges in the Vision and Natural Language domains. We make it easy to recognize text, scan barcodes, track and classify objects in real-time, do translation of text, and more.

The original version of ML Kit was tightly integrated with Firebase, and we heard from many of you that you wanted more flexibility when implementing it in your apps. As a result, we are now making all the on-device APIs available in a new standalone ML Kit SDK that no longer requires a Firebase project. You can still use both ML Kit and Firebase to get the best of both products if you choose to.

With this change, ML Kit is now fully focused on on-device machine learning, giving you access to the unique benefits that on-device versus cloud ML offers:

  • It’s fast, unlocking real-time use cases- since processing happens on the device, there is no network latency. This means, we can do inference on a stream of images / video or multiple times a second on text strings.
  • Works offline - you can rely on our APIs even when the network is spotty or your app’s end-user is in an area without connectivity.
  • Privacy is retained: since all processing is performed locally, there is no need to send sensitive user data over the network to a server.

Naturally, you still get access to Google’s on-device models and processing pipelines, all accessible through easy-to-use APIs, and offered at no cost.

All ML Kit resources can now be found on our new website where we made it a lot easier to access sample apps, API reference docs and our community channels that are there to help you if you have questions.

Object detection & tracking gif Text recognition + Language ID + Translate gif

What does this mean if I already use ML Kit today?

If you are using ML Kit for Firebase’s on-device APIs in your app today, we recommend you to migrate to the new standalone ML Kit SDK to benefit from new features and updates. For more information and step-by-step instructions to update your app, please follow our Migration guide. The cloud-based APIs, model deployment and AutoML Vision Edge remain available through Firebase Machine Learning.

Shrink your app footprint with Google Play Services

Apart from making ML Kit easier to use, developers also asked if we can ship ML Kit through Google Play Services resulting in a smaller app footprint and the model can be reused between apps. Apart from Barcode scanning and Text recognition, we have now added Face detection / contour (model size: 20MB) to the list of APIs that support this functionality. 

// Face detection / Face contour model
// Delivered via Google Play Services outside your app's APK…
implementation 'com.google.android.gms:play-services-mlkit-face-detection:16.0.0'

// …or bundled with your app's APK
implementation 'com.google.mlkit:face-detection:16.0.0'

Jetpack Lifecycle / CameraX support

Android Jetpack Lifecycle support has been added to all APIs. Developers can use addObserver to automatically manage teardown of ML Kit APIs as the app goes through screen rotation or closure by the user / system. This makes CameraX integration easier. With this release, we are also recommending that developers adopt CameraX in their apps due to the ease of integration and image quality improvements (compared to Camera1) on a wide range of devices. 

// ML Kit now supports Lifecycle
val recognizer = TextRecognizer.newInstance()
lifecycle.addObserver(recognizer)

// ...

// Just like CameraX
val camera = cameraProvider.bindToLifecycle( /* lifecycleOwner= */this,
    cameraSelector, previewUseCase, analysisUseCase)

For an overview of all recent changes, check out the release notes for the new SDK.

Codelab of the day - ML Kit x CameraX

To help you get started with the new ML Kit and its support for CameraX, we have created this code lab to Recognize, Identify Language and Translate text. If you have any questions regarding this code lab, please raise them at StackOverflow and tag it with [google-mlkit]. Our team will monitor this.

screenshot of app running

Early access program

Through our early access program, developers have an opportunity to partner with the ML Kit team and get access to upcoming features. Two new APIs are now available as part of this program:

  • Entity Extraction - Detect entities in text & make them actionable. We have support for phone numbers, addresses, payment numbers, tracking numbers, date/time and more.
  • Pose Detection - Low-latency pose detection supporting 33 skeletal points, including hands and feet tracking.

If you are interested, head over to our early access page for details.

pose detection on man jumping rope

Tomorrow - Support for custom models

ML Kit's turn-key solutions are built to help you take common challenges. However, if you needed to have a more tailored solution, one that required custom models, you typically needed to build an implementation from scratch. To help, we are now providing the option to swap out the default Google models with a custom TensorFlow Lite model. We’re starting with the Image Labeling and Object Detection and Tracking APIs, that now support custom image classification models.

Tomorrow, we will dive a bit deeper into how to find or train a TensorFlow Lite model and use it either with ML Kit, or with Android Studio’s new ML binding functionality.

New features to acquire and retain subscribers

Posted by Angela Ying, Product Manager, Google Play

Subscription continues to be one of the fastest growing business models for apps in Google Play. As your subscription business evolves and becomes more sophisticated, our platform continues to evolve to better support your needs. Today we’re excited to tell you more about the new subscription capabilities we announced at the Android 11 Beta Launch, including promotional codes to help you access new subscribers, new opportunities to remind users of your value and win back churned users. Many of these capabilities are built on top of the Play Billing Library version 3.

In addition to the new capabilities, we are also making improvements to our existing platform. Over the past few years, we have launched many features, such as account hold, restore, and pause, which have been highly effective in reducing your voluntary and involuntary churn. We want to ensure that everyone can take advantage of them, which is why we are planning on changing the default settings for these features from optional to either mandatory or on by default starting on November 1, 2020. Additional details on these features and implementation requirements can be found at the end of this post.

Here’s everything that is changing about the subscriptions platform:

Subscriptions platform

More targeted promotions

Promotions and deals are an important way to grow your business to acquire new customers. That’s why over the last year, we have invested in new promotional code capabilities for subscriptions that you can use to send promotions to a more targeted set of users.

Last year at I/O 2019, we launched subscription one-time promotion codes, unique alphanumeric codes that can be distributed to individual users for redemption. Now, we have launched a new frictionless redemption flow which allows users to easily redeem the code, purchase the subscription, and install the app in the Play store in a few simple steps. This greatly simplifies the user experience by reducing the friction users go through to use your code. Since this subscription is started outside of your app, it is only available to developers who are using Billing Library 2.0 or higher.

example mobile displays
In addition to one-time codes, we are excited to officially announce the launch of custom codes (also known as vanity codes), which can be redeemed by multiple users and can be used for marketing campaigns to drive acquisitions. For example, you can post custom codes in advertisements or in social promotions to creatively engage with potential new users. Users can redeem a custom code in your app by entering it in their payment methods when purchasing a subscription.

Remind users of the value of your subscription

Retaining your subscribers is crucial for the long-term health of your subscriptions business. The reason why users stay subscribed is because they perceive ongoing value from your subscription service. To help you communicate this value, we recently launched a module that will remind users of the benefits gained from a subscription when they go to cancel. To take advantage of this module, add a short list of up to 4 subscriber benefits in the Google Play Console.

Win back churned subscribers

If users do churn from their subscription, we want to make it easy for them to restart it whenever they want. To help you do this, we have launched the ability for users to resubscribe to recently expired subscriptions directly from the Google Play subscriptions center. You can enable your SKUs for resubscribe in the Google Play Console. Since this subscription is started outside of your app, it is only available to developers who are using Billing Library version 2 or higher.

Price decreases without opt-ins

Finally, we’ve heard your feedback that requiring users to opt in to subscription price decreases was too restrictive. We are happy to announce that subscription price decreases will no longer require users to take action to opt in to keep their subscription. Users will be notified of an upcoming price decrease and be able to see the upcoming change in the Google Play subscriptions center.

Updated platform retention settings

Over the past few years, our platform has made strides in helping you keep your subscribers, through features aimed at decreasing both voluntary churn and involuntary churn (churn due to payment failure). For example, account hold has helped developers achieve 8% lower involuntary churn and 35% higher payment decline recovery rate compared to developers without account hold. Although these features are effective, retention may not be something that you are thinking about when starting out for the first time.

That’s why we are announcing updated defaults for several subscriptions features that have been up until now optional, which will take effect on November 1, 2020.

  • Account hold and restore will both be mandatory for all developers.
    • Account hold is a state the user enters after a renewal fails due to a payment issue. During account hold, the user loses access to the subscription while Google notifies the user and retries the payment method. Learn how to integrate account hold
    • Restore enables users to resume auto-renewals after they have cancelled the subscription but before the subscription expires. Learn how to integrate restore
  • Pause and resubscribe will be turned on by default for all developers with subscriptions enabled. You can opt-out of either feature in Google Play Console at any time, in case you are unable to implement the changes by November.
    • Pause enables users to pause their subscription for up to 3 months. At the end of the pause period, the subscription will auto-resume. Pause requires Account Hold to be enabled. You can disable the feature by selecting “Disabled” next to Pause in the subscription settings of the Play console. Learn how to integrate pause
    • Resubscribe enables users to resubscribe to a churned subscription within 12 months of subscription expiry. This feature is only available to apps that support Billing Library versions 2.0 and above. You can disable the feature by changing the setting to “Disabled” for each subscription product in the Play console. Learn how to integrate resubscribe

You may have to make modifications to your app or your server to handle these new features. Specifically, your app should:

  • Recognize when a user loses access to the subscription and when the user regains it later
    • If your app relies on the Billing Library and not the Google Play Developer API Purchases.subscriptions to maintain the latest state of your subscriber, then your app should automatically be able to handle this.
    • However, if you rely on the Google Play Developer API, which is common for developers whose subscription is accessible across multiple platforms such as web, it is important that you always have the latest status of the subscriber in your server.
    • To ensure you always have the latest subscriber status, we strongly recommend implementing Real-Time Developer Notifications. Learn more.
  • Gracefully handle out of app purchases (Billing Library 2.0+ only)
    • When a user opens the app after resubscribing, make sure you acknowledge the purchase and show an in-app message recognizing the new purchase. Check out our best practices for handling out of app purchases.

Although not every feature will require you to make engineering changes, we highly recommend that you test each feature before November 1. To make the transition easier, Google has enabled Account Hold, Pause, Restore, and Resubscribe for all license test accounts. Learn more about testing for subscriptions.





How useful did you find this blog post?

★ ★ ★ ★ ★

New features to acquire and retain subscribers

Posted by Angela Ying, Product Manager, Google Play

Subscription continues to be one of the fastest growing business models for apps in Google Play. As your subscription business evolves and becomes more sophisticated, our platform continues to evolve to better support your needs. Today we’re excited to tell you more about the new subscription capabilities we announced at the Android 11 Beta Launch, including promotional codes to help you access new subscribers, new opportunities to remind users of your value and win back churned users. Many of these capabilities are built on top of the Play Billing Library version 3.

In addition to the new capabilities, we are also making improvements to our existing platform. Over the past few years, we have launched many features, such as account hold, restore, and pause, which have been highly effective in reducing your voluntary and involuntary churn. We want to ensure that everyone can take advantage of them, which is why we are planning on changing the default settings for these features from optional to either mandatory or on by default starting on November 1, 2020. Additional details on these features and implementation requirements can be found at the end of this post.

Here’s everything that is changing about the subscriptions platform:

Subscriptions platform

More targeted promotions

Promotions and deals are an important way to grow your business to acquire new customers. That’s why over the last year, we have invested in new promotional code capabilities for subscriptions that you can use to send promotions to a more targeted set of users.

Last year at I/O 2019, we launched subscription one-time promotion codes, unique alphanumeric codes that can be distributed to individual users for redemption. Now, we have launched a new frictionless redemption flow which allows users to easily redeem the code, purchase the subscription, and install the app in the Play store in a few simple steps. This greatly simplifies the user experience by reducing the friction users go through to use your code. Since this subscription is started outside of your app, it is only available to developers who are using Billing Library 2.0 or higher.

example mobile displays
In addition to one-time codes, we are excited to officially announce the launch of custom codes (also known as vanity codes), which can be redeemed by multiple users and can be used for marketing campaigns to drive acquisitions. For example, you can post custom codes in advertisements or in social promotions to creatively engage with potential new users. Users can redeem a custom code in your app by entering it in their payment methods when purchasing a subscription.

Remind users of the value of your subscription

Retaining your subscribers is crucial for the long-term health of your subscriptions business. The reason why users stay subscribed is because they perceive ongoing value from your subscription service. To help you communicate this value, we recently launched a module that will remind users of the benefits gained from a subscription when they go to cancel. To take advantage of this module, add a short list of up to 4 subscriber benefits in the Google Play Console.

Win back churned subscribers

If users do churn from their subscription, we want to make it easy for them to restart it whenever they want. To help you do this, we have launched the ability for users to resubscribe to recently expired subscriptions directly from the Google Play subscriptions center. You can enable your SKUs for resubscribe in the Google Play Console. Since this subscription is started outside of your app, it is only available to developers who are using Billing Library version 2 or higher.

Price decreases without opt-ins

Finally, we’ve heard your feedback that requiring users to opt in to subscription price decreases was too restrictive. We are happy to announce that subscription price decreases will no longer require users to take action to opt in to keep their subscription. Users will be notified of an upcoming price decrease and be able to see the upcoming change in the Google Play subscriptions center.

Updated platform retention settings

Over the past few years, our platform has made strides in helping you keep your subscribers, through features aimed at decreasing both voluntary churn and involuntary churn (churn due to payment failure). For example, account hold has helped developers achieve 8% lower involuntary churn and 35% higher payment decline recovery rate compared to developers without account hold. Although these features are effective, retention may not be something that you are thinking about when starting out for the first time.

That’s why we are announcing updated defaults for several subscriptions features that have been up until now optional, which will take effect on November 1, 2020.

  • Account hold and restore will both be mandatory for all developers.
    • Account hold is a state the user enters after a renewal fails due to a payment issue. During account hold, the user loses access to the subscription while Google notifies the user and retries the payment method. Learn how to integrate account hold
    • Restore enables users to resume auto-renewals after they have cancelled the subscription but before the subscription expires. Learn how to integrate restore
  • Pause and resubscribe will be turned on by default for all developers with subscriptions enabled. You can opt-out of either feature in Google Play Console at any time, in case you are unable to implement the changes by November.
    • Pause enables users to pause their subscription for up to 3 months. At the end of the pause period, the subscription will auto-resume. Pause requires Account Hold to be enabled. You can disable the feature by selecting “Disabled” next to Pause in the subscription settings of the Play console. Learn how to integrate pause
    • Resubscribe enables users to resubscribe to a churned subscription within 12 months of subscription expiry. This feature is only available to apps that support Billing Library versions 2.0 and above. You can disable the feature by changing the setting to “Disabled” for each subscription product in the Play console. Learn how to integrate resubscribe

You may have to make modifications to your app or your server to handle these new features. Specifically, your app should:

  • Recognize when a user loses access to the subscription and when the user regains it later
    • If your app relies on the Billing Library and not the Google Play Developer API Purchases.subscriptions to maintain the latest state of your subscriber, then your app should automatically be able to handle this.
    • However, if you rely on the Google Play Developer API, which is common for developers whose subscription is accessible across multiple platforms such as web, it is important that you always have the latest status of the subscriber in your server.
    • To ensure you always have the latest subscriber status, we strongly recommend implementing Real-Time Developer Notifications. Learn more.
  • Gracefully handle out of app purchases (Billing Library 2.0+ only)
    • When a user opens the app after resubscribing, make sure you acknowledge the purchase and show an in-app message recognizing the new purchase. Check out our best practices for handling out of app purchases.

Although not every feature will require you to make engineering changes, we highly recommend that you test each feature before November 1. To make the transition easier, Google has enabled Account Hold, Pause, Restore, and Resubscribe for all license test accounts. Learn more about testing for subscriptions.





How useful did you find this blog post?

★ ★ ★ ★ ★

Announcing the winners of the #AndroidDevChallenge, powered by on-device machine learning

Posted by Jacob Lehrbaum, Director of Developer Relations, Android

Developers like you have always played an important role in Android innovation. Over 10 years ago, when we first launched the Android SDK, we also announced the Android Developer Challenge to reward model apps and highlight new ways of solving user problems. As Android pushes the boundaries of machine learning, 5G, foldables, and more, developers continue to help shape these new frontiers. To celebrate this work, we revived the challenge in 2019, with a focus on “Helpful Innovation,” powered by on-device machine learning.

We received hundreds of creative projects, and at the end of last year, picked 10 winners who each combined a strong idea and a thirst to bring it to life. Since then, we’ve been working with those winners to help turn their ideas into reality. And today, we’re announcing the 10 winners. Some are still at the beginning of their journey but but their apps are now ready for you to download and try out! !

  • AgroDoc helps farmers diagnose plant disease and make treatment plans. [Navneet Krishna; Kochi, India]
  • AgriFarm helps farmers detect plant diseases and prevent major damage in fruits and vegetables such as tomatoes, corn and potatoes. [Balochisan, Pakistan]
  • Eskke streamlines mobile money management for people in the Congo, letting them transfer money, pay bills, buy subscriptions and essential airtime through SMS. [David Mumbere Kathoh; Goma, Democratic Republic of Congo]
  • Leepi helps students learn hand gestures and symbols for American Sign Language. [Prince Patel; Bengaluru, India]
  • MixPose is a live streaming platform that gives yoga teachers and fitness professionals the opportunity to teach, track alignment, and give feedback in real-time. [Peter Ma; San Francisco, California, USA]
  • Pathfinder could help people with visual impairments navigate complex situations by identifying and calculating the trajectories of objects moving in their path. [Colin Shelton; Addison, Texas, USA]
  • Snore & Cough helps you identify and analyze snoring and coughing, to help provide info to users seeking assistance from a medical professional. [Ethan Fan; Mountain View, California, USA]
  • Stila pairs with a wearable device, like the Fitbit wristband or a device running on Wear OS by Google to monitor and track the body’s stress levels. By monitoring stress levels over time, you have the chance to better understand and manage stress in your life. [Yingdin Wing; Munich, Germany]
  • Trashly makes recycling easier. Just point the on-device camera at an item, and through object detection, the app identifies and classifies plastic and paper cups, bags, bottles, etc. [Elvin Rakhmankulov; Chicago, Illinois, USA]
  • UnoDogs helps owners better support their pet’s wellness, providing customized information and fitness programs. [Chinmany Mishra; New Delhi, India]

Making on-device machine learning more accessible, with ML Kit and TensorFlow Lite

Increasingly, machine learning is becoming a more accessible tool to developers with limited to no background in the technology. In fact, for most of the winners of the Android Developer Challenge, this was their first foray into machine learning. That’s thanks in part to two key offerings from Google, which bring on-device machine learning into reach for millions of developers around the world.

The first is ML Kit. ML Kit brings Google’s on-device machine learning technologies to mobile app developers, so they can build customized and interactive experiences into their apps. This includes tools such as language translation, text recognition, object detection and more. Eskke, for instance, uses offline text recognition and barcode scanning from ML Kit so users can scan the QR code at a mobile money kiosk and quickly withdraw money. And MixPose uses ML Kit's forthcoming Pose detection API to detect each user’s yoga positions and movements, so teachers can provide feedback.

The other Google resource that many of the Android Dev Challenge winners used was TensorFlow Lite. This powerful machine learning framework can help run machine learning models on Android, iOS and IoT devices that would never normally be able to support them. Its set of tools can be used for all kinds of powerful neural network-related applications, from image detection to speech recognition, bringing the latest cutting-edge technology to the devices we carry around with us wherever we go. Trashly, for instance, uses a custom TensorFlow Lite model to report if an object is recyclable and how to recycle it.

Helpful innovation, such as the 10 winning apps in the Android Developer Challenge, has the potential to change the way we access, use, and interpret information, making it available when we need it, where we need it most. By working with these developers focused on helpful innovation, we hope to inspire the next wave of developers to unlock what’s possible with this new technology.

#11WeeksOfAndroid Week 2 Machine Learning with Android logo head

What’s next in Android Machine Learning week?

As we kick off the second week of #11WeeksOfAndroid, focused on Machine Learning, we will highlight new tools and resources available to Android developers. Here’s a taste of the rest of this week:

  • Tuesday - ML Kit, the turnkey ML SDK went through a major overhaul with its new on-device offering this month. Check out the substantial improvement in developer usability, CameraX support and where the platform is going next.
  • Wednesday - Custom Models. When prepackaged SDK doesn’t quite satisfy your need, tools from Android Studio, TensorFlow Lite and ML Kit might just be the answer. Aside from individual offerings, we will also highlight how they can be used together.
  • Thursday - ML design. Learn some best practices for making ML product decisions from the People + AI Guidebook. We will go behind the scenes of the Read Along app, an on-device ML app that helps grow universal literacy. Bring your whole team because everyone, including UXers, engineers, and product managers are invited!

On Tuesday and Wednesday, we will also have a “codelab of the day” so get your Android Studio 4.1 beta today, block off an hour in your schedule and take this ML journey with us!

*The apps presented here are the projects of the developers individually, and not Google.

Announcing the winners of the #AndroidDevChallenge, powered by on-device machine learning

Posted by Jacob Lehrbaum, Director of Developer Relations, Android

Developers like you have always played an important role in Android innovation. Over 10 years ago, when we first launched the Android SDK, we also announced the Android Developer Challenge to reward model apps and highlight new ways of solving user problems. As Android pushes the boundaries of machine learning, 5G, foldables, and more, developers continue to help shape these new frontiers. To celebrate this work, we revived the challenge in 2019, with a focus on “Helpful Innovation,” powered by on-device machine learning.

We received hundreds of creative projects, and at the end of last year, picked 10 winners who each combined a strong idea and a thirst to bring it to life. Since then, we’ve been working with those winners to help turn their ideas into reality. And today, we’re announcing the 10 winners. Some are still at the beginning of their journey but but their apps are now ready for you to download and try out! !

  • AgroDoc helps farmers diagnose plant disease and make treatment plans. [Navneet Krishna; Kochi, India]
  • AgriFarm helps farmers detect plant diseases and prevent major damage in fruits and vegetables such as tomatoes, corn and potatoes. [Balochisan, Pakistan]
  • Eskke streamlines mobile money management for people in the Congo, letting them transfer money, pay bills, buy subscriptions and essential airtime through SMS. [David Mumbere Kathoh; Goma, Democratic Republic of Congo]
  • Leepi helps students learn hand gestures and symbols for American Sign Language. [Prince Patel; Bengaluru, India]
  • MixPose is a live streaming platform that gives yoga teachers and fitness professionals the opportunity to teach, track alignment, and give feedback in real-time. [Peter Ma; San Francisco, California, USA]
  • Pathfinder could help people with visual impairments navigate complex situations by identifying and calculating the trajectories of objects moving in their path. [Colin Shelton; Addison, Texas, USA]
  • Snore & Cough helps you identify and analyze snoring and coughing, to help provide info to users seeking assistance from a medical professional. [Ethan Fan; Mountain View, California, USA]
  • Stila pairs with a wearable device, like the Fitbit wristband or a device running on Wear OS by Google to monitor and track the body’s stress levels. By monitoring stress levels over time, you have the chance to better understand and manage stress in your life. [Yingdin Wing; Munich, Germany]
  • Trashly makes recycling easier. Just point the on-device camera at an item, and through object detection, the app identifies and classifies plastic and paper cups, bags, bottles, etc. [Elvin Rakhmankulov; Chicago, Illinois, USA]
  • UnoDogs helps owners better support their pet’s wellness, providing customized information and fitness programs. [Chinmany Mishra; New Delhi, India]

Making on-device machine learning more accessible, with ML Kit and TensorFlow Lite

Increasingly, machine learning is becoming a more accessible tool to developers with limited to no background in the technology. In fact, for most of the winners of the Android Developer Challenge, this was their first foray into machine learning. That’s thanks in part to two key offerings from Google, which bring on-device machine learning into reach for millions of developers around the world.

The first is ML Kit. ML Kit brings Google’s on-device machine learning technologies to mobile app developers, so they can build customized and interactive experiences into their apps. This includes tools such as language translation, text recognition, object detection and more. Eskke, for instance, uses offline text recognition and barcode scanning from ML Kit so users can scan the QR code at a mobile money kiosk and quickly withdraw money. And MixPose uses ML Kit's forthcoming Pose detection API to detect each user’s yoga positions and movements, so teachers can provide feedback.

The other Google resource that many of the Android Dev Challenge winners used was TensorFlow Lite. This powerful machine learning framework can help run machine learning models on Android, iOS and IoT devices that would never normally be able to support them. Its set of tools can be used for all kinds of powerful neural network-related applications, from image detection to speech recognition, bringing the latest cutting-edge technology to the devices we carry around with us wherever we go. Trashly, for instance, uses a custom TensorFlow Lite model to report if an object is recyclable and how to recycle it.

Helpful innovation, such as the 10 winning apps in the Android Developer Challenge, has the potential to change the way we access, use, and interpret information, making it available when we need it, where we need it most. By working with these developers focused on helpful innovation, we hope to inspire the next wave of developers to unlock what’s possible with this new technology.

#11WeeksOfAndroid Week 2 Machine Learning with Android logo head

What’s next in Android Machine Learning week?

As we kick off the second week of #11WeeksOfAndroid, focused on Machine Learning, we will highlight new tools and resources available to Android developers. Here’s a taste of the rest of this week:

  • Tuesday - ML Kit, the turnkey ML SDK went through a major overhaul with its new on-device offering this month. Check out the substantial improvement in developer usability, CameraX support and where the platform is going next.
  • Wednesday - Custom Models. When prepackaged SDK doesn’t quite satisfy your need, tools from Android Studio, TensorFlow Lite and ML Kit might just be the answer. Aside from individual offerings, we will also highlight how they can be used together.
  • Thursday - ML design. Learn some best practices for making ML product decisions from the People + AI Guidebook. We will go behind the scenes of the Read Along app, an on-device ML app that helps grow universal literacy. Bring your whole team because everyone, including UXers, engineers, and product managers are invited!

On Tuesday and Wednesday, we will also have a “codelab of the day” so get your Android Studio 4.1 beta today, block off an hour in your schedule and take this ML journey with us!

*The apps presented here are the projects of the developers individually, and not Google.

11 Weeks of Android: People & Identity

Posted by Dr. Stefan Frank, Senior Product Manager, Android System UI

#11 weeks of Android with Android logo

This blog post is part of a weekly series for #11WeeksOfAndroid. Each week we’re diving into a key area of Android so you don’t miss anything. This week, we spotlight people & identity. Here's a look at what you should know.

The big news

One of the goals of Android 11 was making our phones more people-centric. Because nothing matters more to people than connecting to loved ones. It is a core human need especially during our current physical distancing constraints. We have the need to be more social than ever before. Android 11 reimagines how we have conversations on our phones by adding new capabilities to help you maintain your identity across multiple devices.

We are announcing some new features in Android 11 that allows you to easily connect with your loved ones, friends and business colleagues. At the center of this release is the Android Conversation Shortcut API and Identity Services Library. These new tools empower instant connections to your best friend, sharing funny pictures of your dogs, telling your aunt about a tasty seafood recipe that you discovered, or congratulating an office colleague about her promotion. And they also provide a new level of password management that makes it easier for your users to sign up and sign in.

One of our favorite features brings conversations from people that matter most to you right to the lock screen of your phone. You’ll easily recognize them by their avatar and instantly respond to your family, friends, or colleagues. These are people you truly want to connect to. We knew this feature would be useful, but the responses from our beta-testers made us smile. The decision to include the Conversation Shortcut API to improve the lives of our users was one of the easiest decisions for us to make in the release of Android 11.

Creating a bubble from an incoming notification and accessing the conversation from the bubble.

Creating a bubble from an incoming notification and accessing the conversation from the bubble.

One of the new features building on top of shortcuts is the new conversation space at the top of your notifications. It focuses your attention on what matters most - your conversations. Right from these notifications the user can trigger another new feature in Android 11 - Bubbles. Bubbles are small representations of conversations floating over other content on the side of the screen, which can be expanded to allow quick access to conversations, without changing what you were doing on the device. They are super handy for carrying on conversations while using the device for other tasks.

The new conversations space showing how a conversation is marked as priority and will be displayed on the lock screen.

The new conversations space showing how a conversation is marked as priority and will be displayed on the lock screen.

A long tap on conversation notifications enables the user to mark priority conversations in order to give special prominence to the most important people. Priority conversations will be displayed with their individual avatar right on the lock screen and move to the top of your notifications. They can even be set to break through do-not-disturb. Another use of the conversation shortcuts is that they are used as share targets in the system share sheet, which was already launched in Android 10.

Another focus of this week was identity. To tackle user and developer complexity that makes identity a challenge for developers, we've been working on One Tap and Block Store, part of our new Google Identity Services Library. One Tap is our new cross-platform sign-in mechanism for Web and Android, supporting and streamlining multiple types of credentials. Block Store is our new token-based sign-in mechanism that’s built on top of Backup and Restore. It allows you to keep your user signed in across Android devices.

We’re super excited about all these features [since] they help all of us connect, communicate and express ourselves to the people that we care about and to the apps we are dealing with -- which is as important now as it’s ever been.”

What to watch

For a high level overview of the people centric functionality, we recommend that you check out the Android 11 launch highlight video on People. Earlier this week, we also launched a new talk on ‘Conversation Notifications’, where Artur describes how to implement the conversation shortcut and bubbles. There is also a great overview talk on the conversation additions and other System UI news from Dan. Finally, you can also listen to Chet’s podcasts where he interviews us on People and Bubbles.

If you’re interested in learning more about Identity, we also published “In Identity on Android: What’s new in sign-in”, this week. In this video, Vishal explains the new libraries in the Google Identity System: One Tap and Block Store."

Two of the teams that worked very early with us on these conversation specific topics are the Messenger team from Facebook and the direct messaging team from Twitter. Read the stories around both of these implementations here and here.

Learning path

If you’re looking for an easy way to pick up the highlights of this week, check out the People and Identity pathway. A pathway is an ordered tutorial that allows users to complete a pre-defined module that culminates in a quiz. It may include codelabs, videos, articles and blog posts. A virtual badge is awarded to each user who passes the quiz. Test your knowledge of key takeaways about People and Identity to earn a limited edition badge.

Key takeaways

Android 11 is the starting point of an ongoing focus on what matters most to users, people and conversations. Many of our partners in our ecosystem introduced amazing apps and services enabling these connections with people and conversations. We in Android want to elevate and surface these partners more prominently in order to support this goal. Thus if you are working on an app that fosters real time communication between people, we strongly encourage you to adopt the conversation shortcut based APIs for notifications, bubbles and sharing when targeting API 30 in order to put users’ conversations front and center and give them quick access to your app. The developer documentation can be found here.

For apps that handle user accounts, we encourage you to help our users avoid messy password hunting and forgotten credential processes by integrating One Tap to streamline credential management and Block Store to handle device updates. These integrations will work on phones back to Android M.

In the spirit of this week, I wish you meaningful and joyful connections with the people that matter to you and seamless experiences with your favorite apps. We hope you will help us in our journey of supporting these goals.

Resources

You can find the entire playlist of #11WeeksOfAndroid video content here, and learn more about each week here. We’ll continue to spotlight new areas each week, so keep an eye out and follow us on Twitter and YouTube. Thanks so much for letting us be a part of this experience with you!

11 Weeks of Android: People & Identity

Posted by Dr. Stefan Frank, Senior Product Manager, Android System UI

#11 weeks of Android with Android logo

This blog post is part of a weekly series for #11WeeksOfAndroid. Each week we’re diving into a key area of Android so you don’t miss anything. This week, we spotlight people & identity. Here's a look at what you should know.

The big news

One of the goals of Android 11 was making our phones more people-centric. Because nothing matters more to people than connecting to loved ones. It is a core human need especially during our current physical distancing constraints. We have the need to be more social than ever before. Android 11 reimagines how we have conversations on our phones by adding new capabilities to help you maintain your identity across multiple devices.

We are announcing some new features in Android 11 that allows you to easily connect with your loved ones, friends and business colleagues. At the center of this release is the Android Conversation Shortcut API and Identity Services Library. These new tools empower instant connections to your best friend, sharing funny pictures of your dogs, telling your aunt about a tasty seafood recipe that you discovered, or congratulating an office colleague about her promotion. And they also provide a new level of password management that makes it easier for your users to sign up and sign in.

One of our favorite features brings conversations from people that matter most to you right to the lock screen of your phone. You’ll easily recognize them by their avatar and instantly respond to your family, friends, or colleagues. These are people you truly want to connect to. We knew this feature would be useful, but the responses from our beta-testers made us smile. The decision to include the Conversation Shortcut API to improve the lives of our users was one of the easiest decisions for us to make in the release of Android 11.

Creating a bubble from an incoming notification and accessing the conversation from the bubble.

Creating a bubble from an incoming notification and accessing the conversation from the bubble.

One of the new features building on top of shortcuts is the new conversation space at the top of your notifications. It focuses your attention on what matters most - your conversations. Right from these notifications the user can trigger another new feature in Android 11 - Bubbles. Bubbles are small representations of conversations floating over other content on the side of the screen, which can be expanded to allow quick access to conversations, without changing what you were doing on the device. They are super handy for carrying on conversations while using the device for other tasks.

The new conversations space showing how a conversation is marked as priority and will be displayed on the lock screen.

The new conversations space showing how a conversation is marked as priority and will be displayed on the lock screen.

A long tap on conversation notifications enables the user to mark priority conversations in order to give special prominence to the most important people. Priority conversations will be displayed with their individual avatar right on the lock screen and move to the top of your notifications. They can even be set to break through do-not-disturb. Another use of the conversation shortcuts is that they are used as share targets in the system share sheet, which was already launched in Android 10.

Another focus of this week was identity. To tackle user and developer complexity that makes identity a challenge for developers, we've been working on One Tap and Block Store, part of our new Google Identity Services Library. One Tap is our new cross-platform sign-in mechanism for Web and Android, supporting and streamlining multiple types of credentials. Block Store is our new token-based sign-in mechanism that’s built on top of Backup and Restore. It allows you to keep your user signed in across Android devices.

We’re super excited about all these features [since] they help all of us connect, communicate and express ourselves to the people that we care about and to the apps we are dealing with -- which is as important now as it’s ever been.”

What to watch

For a high level overview of the people centric functionality, we recommend that you check out the Android 11 launch highlight video on People. Earlier this week, we also launched a new talk on ‘Conversation Notifications’, where Artur describes how to implement the conversation shortcut and bubbles. There is also a great overview talk on the conversation additions and other System UI news from Dan. Finally, you can also listen to Chet’s podcasts where he interviews us on People and Bubbles.

If you’re interested in learning more about Identity, we also published “In Identity on Android: What’s new in sign-in”, this week. In this video, Vishal explains the new libraries in the Google Identity System: One Tap and Block Store."

Two of the teams that worked very early with us on these conversation specific topics are the Messenger team from Facebook and the direct messaging team from Twitter. Read the stories around both of these implementations here and here.

Learning path

If you’re looking for an easy way to pick up the highlights of this week, check out the People and Identity pathway. A pathway is an ordered tutorial that allows users to complete a pre-defined module that culminates in a quiz. It may include codelabs, videos, articles and blog posts. A virtual badge is awarded to each user who passes the quiz. Test your knowledge of key takeaways about People and Identity to earn a limited edition badge.

Key takeaways

Android 11 is the starting point of an ongoing focus on what matters most to users, people and conversations. Many of our partners in our ecosystem introduced amazing apps and services enabling these connections with people and conversations. We in Android want to elevate and surface these partners more prominently in order to support this goal. Thus if you are working on an app that fosters real time communication between people, we strongly encourage you to adopt the conversation shortcut based APIs for notifications, bubbles and sharing when targeting API 30 in order to put users’ conversations front and center and give them quick access to your app. The developer documentation can be found here.

For apps that handle user accounts, we encourage you to help our users avoid messy password hunting and forgotten credential processes by integrating One Tap to streamline credential management and Block Store to handle device updates. These integrations will work on phones back to Android M.

In the spirit of this week, I wish you meaningful and joyful connections with the people that matter to you and seamless experiences with your favorite apps. We hope you will help us in our journey of supporting these goals.

Resources

You can find the entire playlist of #11WeeksOfAndroid video content here, and learn more about each week here. We’ll continue to spotlight new areas each week, so keep an eye out and follow us on Twitter and YouTube. Thanks so much for letting us be a part of this experience with you!

Messenger and Conversations

Facebook logo

This blogpost is a collaboration between Google and Messenger from Facebook. Authored by Aaron Labiaga with support from Caleb Gomer and Samuel Guirado from Messenger.


Messenger is ubiquitous in the messaging app world and has pioneered the floating chat bubble. Bubbles help users keep conversations in view and accessible while multitasking, overlaying other UI elements in the foreground, and providing users easy access and visibility to their ongoing chats. Bubbles is one way Android 11 is making the platform more people-centric and expressive, reimagining the way we have conversations on our phones. Messenger’s early pioneering of the floating chat bubble, and the strong reception by users, helped lead to its native implementation in the framework.

new conversations ui gif

Bubbles

The Bubbles API is built on top of the notifications API and is exclusively focused on people in Android 11. First Introduced in Android 10, what was previously an opt-in feature is now on by default. To use bubbles, the developer must create BubbleMetadata, which is set on the notification. This metadata describes the Activity to launch when a bubble is clicked, along with various behaviors relevant to the expanded bubble. The Activity must follow the criteria of being embeddable and resizable in order to use it in a bubble.

Notification bubbles are reserved for conversations with persons in context. These are MessagingStyle notifications with a set long-lived shortcut ID. Please see the following Bubbles code sample how.


A Q&A with the Messenger team

The Messenger team shares their experience with the migration and prospect of the impact of the changes.

How was the migration to bubbles, technical challenges, scope, and impact on codebase?

Prior to Bubbles, Messenger used the SYSTEM_ALERT_WINDOW for its implementation of Bubbles. It achieved our purpose, but hosting complex Android UI outside of Activities is challenging to implement and maintain. Using this natively supported API allowed us to build more traditional, Activity-based Android UI that works well in Bubbles and full screen. This new Bubbles-based chat experience is much simpler and more maintainable than our SAW-based one. We are excited that Android believes that it is a user experience that will help drive improvements in the conversation space.
Ensuring that bubble shortcuts were up-to-date with the latest state of the conversation thread was a technical challenge worth noting. Picture changed, conversation deleted or contact blocked are events that require the bubble shortcut to be updated or even deleted. The Shortcuts API allows for easily registering/unregistering shortcuts and for querying and updating existing ones, which made this whole process very straight forward.

What are your future prospects on the impact of Messenger messages in the conversation space?

The conversation section will give our messages the right visibility. Given that conversation section ranks high in the notification drawer, we definitely want to be present in that space.

A people-centric experience in Android 11

Bubbles are just one way that Android 11 puts people at the heart of the experience for Android; if you’re a messaging or chat app, you should consider using the Bubbles API to help your users as they multi-task. It’s great to see apps like Messenger navigate the openness of Android to create innovative new experiences, and we’re excited to make Bubbles a native experience in Android 11. For more information, please visit the Conversation API guidelines.

Messenger and Conversations

Facebook logo

This blogpost is a collaboration between Google and Messenger from Facebook. Authored by Aaron Labiaga with support from Caleb Gomer and Samuel Guirado from Messenger.


Messenger is ubiquitous in the messaging app world and has pioneered the floating chat bubble. Bubbles help users keep conversations in view and accessible while multitasking, overlaying other UI elements in the foreground, and providing users easy access and visibility to their ongoing chats. Bubbles is one way Android 11 is making the platform more people-centric and expressive, reimagining the way we have conversations on our phones. Messenger’s early pioneering of the floating chat bubble, and the strong reception by users, helped lead to its native implementation in the framework.

new conversations ui gif

Bubbles

The Bubbles API is built on top of the notifications API and is exclusively focused on people in Android 11. First Introduced in Android 10, what was previously an opt-in feature is now on by default. To use bubbles, the developer must create BubbleMetadata, which is set on the notification. This metadata describes the Activity to launch when a bubble is clicked, along with various behaviors relevant to the expanded bubble. The Activity must follow the criteria of being embeddable and resizable in order to use it in a bubble.

Notification bubbles are reserved for conversations with persons in context. These are MessagingStyle notifications with a set long-lived shortcut ID. Please see the following Bubbles code sample how.


A Q&A with the Messenger team

The Messenger team shares their experience with the migration and prospect of the impact of the changes.

How was the migration to bubbles, technical challenges, scope, and impact on codebase?

Prior to Bubbles, Messenger used the SYSTEM_ALERT_WINDOW for its implementation of Bubbles. It achieved our purpose, but hosting complex Android UI outside of Activities is challenging to implement and maintain. Using this natively supported API allowed us to build more traditional, Activity-based Android UI that works well in Bubbles and full screen. This new Bubbles-based chat experience is much simpler and more maintainable than our SAW-based one. We are excited that Android believes that it is a user experience that will help drive improvements in the conversation space.
Ensuring that bubble shortcuts were up-to-date with the latest state of the conversation thread was a technical challenge worth noting. Picture changed, conversation deleted or contact blocked are events that require the bubble shortcut to be updated or even deleted. The Shortcuts API allows for easily registering/unregistering shortcuts and for querying and updating existing ones, which made this whole process very straight forward.

What are your future prospects on the impact of Messenger messages in the conversation space?

The conversation section will give our messages the right visibility. Given that conversation section ranks high in the notification drawer, we definitely want to be present in that space.

A people-centric experience in Android 11

Bubbles are just one way that Android 11 puts people at the heart of the experience for Android; if you’re a messaging or chat app, you should consider using the Bubbles API to help your users as they multi-task. It’s great to see apps like Messenger navigate the openness of Android to create innovative new experiences, and we’re excited to make Bubbles a native experience in Android 11. For more information, please visit the Conversation API guidelines.