Posted by Laurence Moroney, TensorFlow Developer Advocate at Google

On October 16-18, thousands of developers from all over the world are coming together for DevFest 2020, the largest virtual weekend of community-led learning on Google technologies.

For DevFest this year, a few familiar faces from Google and the community came together to show you how to build an app using multiple Google Developer tools to detect crop diseases, from scratch, in just a few minutes. This is one example of how developers can leverage a number of Google tools to solve a real-world problem. Watch the full demo video here or learn more below.

Creating the Android app

Chet Haase, Android Developer Advocate, begins by creating an Android app that recognizes information about plants. To do that, he needs camera functionality, and also machine learning inference.

The app is written in Kotlin, uses CameraX to take the pictures and MLKit for on-device Machine Learning analysis. The core functionality revolves around taking a picture, analyzing it, and displaying the results.

[Code showing how the app takes a picture, analyzes it, and displays the results.]

MLKIt makes it easy to recognize the contents of an image using its ImageLabeler object, so Chet just grabs a frame from CameraX and uses that. When this succeeds, we receive a collection of ImageLabels, which we turn into text strings and display a toast with the results.

[Demo of what the app detecting that the image is a plant.]

Setting up the Machine Learning model

To dig a little deeper, Gus Martins, Google Developer Advocate for TensorFlow, shows us how to set up a Machine Learning model to detect diseases in bean plants.

Gus uses Google Colab, a cloud-hosted development tool to do transfer learning from an existing ML model hosted on TensorFlow.Hub

He then puts it all together and uses a tool called Tensorflow Lite Model Maker to train the model using our custom dataset.

Setting up the Android app to recognize and build classes

The Model Gus created includes all the metadata needed for Android Studio to recognize it and build classes from it that can run inference on the model using TensorFlow Lite. To do so, Annyce Davis, Google Developer Expert for Android, updates the app to use TensorFlow Lite.

She uses the model with an image from the camera to get an inference about a bean leaf to see if it is diseased or not.

Now, when we run our app, instead of telling us it’s looking at a leaf, it can tell us if our bean is healthy or, if not, can give us a diagnosis.

(Demo of the app detecting whether or not the plant is healthy)

Transforming the demo into a successful app using Firebase, Design, and Responsible AI principles

This is just a raw demo. But to transform it into a successful app, Todd Kerpelman, Google Developer Advocate for Firebase, suggests using the Firebase plugin for Android Studio to add some Analytics, so we can find out exactly how our users are interacting with our app.

There's a lot of ways to get at this data -- it will start showing up in the Firebase dashboard, but one really fun way of viewing this data is to use StreamView, which gives you a real-time sample of what kinds of analytics results we're seeing.

[Firebase Streamview allows you to view real-time analytics.]

Using Firebase, you could also, for example, add A/B testing to your app to choose the best model for your users; have remote configuration to keep your app up to date; have easy sign-in to your app if you want users to log in, and a whole lot more!

Di Dang, UX Designer & Design Advocate, reminds us that if we were to productize this app, it’s important to keep in mind how our AI design decisions impact users.

For instance, we need to consider if and/or how it makes sense to display confidence intervals. Or consider how you design the onboarding experience to set user expectations for the capabilities and limitations of your ML-based app, which is vital to app adoption and engagement. For more guidance on AI design decisions, check out the People + AI Guidebook.

[You can learn more about AI design decisions at the People & AI Guidebook]

This use case focuses on plant diseases, but for this case and others, where our ML-based predictions intersect with people or communities, we absolutely need to think about responsible AI themes like privacy and fairness. Learn more here.

Building a Progressive Web App

Paul Kinlan, Developer Advocate for Web, reminds us to not forget about the web!

Paul shows us how to build a PWA that allows users to install an app across all platforms, which can combine the camera with TensorFlow.js to integrate Machine Learning to build an amazing experience that runs in the browser - no additional download required.

After setting up the project with a standard layout (with an HTML file, manifest, and Service Worker to make it a PWA) and a data folder that contains our TensorFlow configuration, we’ll wait until all of the JS and CSS has loaded in order to initialize the app. We then set up the camera with our helper object, and load the TensorFlow model. After it becomes active, we can then set up the UI.

The PWA is now ready and waiting for us to use.

(The PWA tells us whether or not the plant is healthy - no app download necessary!)

The importance of Open Source

And finally, Puuja Rajan, Google Developer Expert for TensorFlow and Women Techmakers lead, reminds us that we might also want to open source this project, too, so that developers can suggest improvements, optimizations and even additional features by filing an issue or sending a pull request. It’s a great way to get your hard work in front of even more people. You can learn more about starting an Open Source project here.

In fact, we’ve already open sourced this project, which you can find here.

So now you have the platform for building a real app -- with the tooling from Android Studio, CameraX, Jetpack, ML Kit, Colab, TensorFlow, Firebase, Chrome and Google Cloud, you have a lot of things that just work better together. This isn’t a finished project by any means, just a proof of concept for how a minimum viable product with a roadmap to completion can be put together using Google’s Developer Tools.

Join us online this weekend at a DevFest near you. Sign up here.

Posted by Wesley Chun, Developer Advocate, Google Cloud

File backup isn't the most exciting topic while analyzing images with AI/ML is more interesting, so combining them probably isn't a workflow you think about often. However, by augmenting the former with the latter, you can build a more useful solution than without. Google provides a diverse array of developer tools you can use to realize this ambition, and in fact, you can craft such a workflow with Google Cloud products alone. More compellingly, the basic principle of mixing-and-matching Google technologies can be applied to many other challenges faced by you, your organization, or your customers.

The sample app presented uses Google Drive and Sheets plus Cloud Storage and Vision to make it happen. The use-case: Google Workspace (formerly G Suite) users who work in industries like architecture or advertising, where multimedia files are constantly generated. Every client job results in yet another Drive subfolder and collection of asset files. Successive projects lead to even more files and folders. At some point, your Drive becomes a "hot mess," making users increasingly inefficient, requiring them to scroll endlessly to find what they're looking for.

A user and their Google Drive files

How can Google Cloud help? Like Drive, Cloud Storage provides file (and generic blob) storage in the cloud. (More on the differences between Drive & Cloud Storage can be found in this video.)

Cloud Storage provides several storage classes depending on how often you expect to access your archived files. The less often files are accessed, the "colder" the storage, and the lower the cost. As users progress from one project to another, they're not as likely to need older Drive folders and those make great candidates to backup to Cloud Storage.

First challenge: determine the security model. When working with Google Cloud APIs, you generally select OAuth client IDs to access data owned by users and service accounts for data owned by applications/projects. The former is typically used with Workspace APIs while the latter is the primary way to access Google Cloud APIs. Since we're using APIs from both product groups, we need to make a decision (for now and change later if desired).

Since the goal is a simple proof-of-concept, user auth suffices. OAuth client IDs are standard for Drive & Sheets API access, and the Vision API only needs API keys so the more-secure OAuth client ID is more than enough. The only IAM permissions to acquire are for the user running the script to get write access to the destination Cloud Storage bucket. Lastly, Workspace APIs don't have their own product client libraries (yet), so the lower-level Google APIs "platform" client libraries serve as a "lowest common denominator" to access all four REST APIs. Those who have written Cloud Storage or Vision code using the Cloud client libraries will see something different.

The prototype is a command-line script. In real life, it would likely be an application in the cloud, executing as a Cloud Function or a Cloud Task running as determined by Cloud Scheduler. In that case, it would use a service account with Workspace domain-wide delegation to act on behalf of an employee to backup their files. See this page in the documentation describing when you'd use this type of delegation and when not to.

Our simple prototype targets individual image files, but you can continue to evolve it to support multiple files, movies, folders, and ZIP archives if desired. Each function calls a different API, creating a "service pipeline" with which to process the images. The first pair of functions are drive_get_file() and gcs_blob_upload(). The former queries for the image on Drive, grabs pertinent metadata (filename, ID, MIMEtype, size), downloads the binary "blob" and returns all of that to the caller. The latter uploads the binary along with relevant metadata to Cloud Storage. The script was written in Python for brevity, but the client libraries support most popular languages. Below is the aforementioned function pseudocode:

def drive_get_file(fname):
    rsp = DRIVE.files().list(q="name='%s'" % fname).execute().get['files'][0]
    fileId, fname, mtype = rsp['id'], rsp['name'], rsp['mimeType']
    blob = DRIVE.files().get_blob(fileId).execute()
    return fname, mtype, rsp['modifiedTime'], blob

def gcs_blob_upload(fname, folder, bucket, blob, mimetype):
    body = {'name': folder+'/'+fname, 'uploadType': 'multipart',
            'contentType': mimetype}
    return GCS.objects().insert(bucket, body, blob).execute()

Next, vision_label_img() passes the binary to the Vision API and formats the results. Finally that information along with the file's archived Cloud Storage location are written as a single row of data in a Google Sheet via sheet_append_roww().

def vision_label_img(img):
    body = {'requests': [{'image': {'content': img},
        'features': [{'type': 'LABEL_DETECTION'}]}]}
    rsp = VISION.images().annotate(body=body).execute().get['responses'][0]
    return ', '.join('(%.2f%%) %s' % (label['score']*100.,
            label['description']) for label in rsp['labelAnnotations'])

def sheet_append_row(sheet_id, row):
    rsp = SHEETS.spreadsheets().values().append(spreadsheetId=sheet_id,
        range='Sheet1', body={'values': row}).execute()
    return rsp.get('updates').get('updatedCells')

Finally, a "main" program that drives the workflow is needed. It comes with a pair of utility functions, _k_ize() to turn file sizes into kilobytes and _linkify() to build a valid Cloud Storage hyperlink as a spreadsheet formula. These are featured here:

def _k_ize(nbytes):  # bytes to KBs (not KiBs) as str
    return '%6.2fK' % (nbytes/1000.)


def _linkify(bucket, fname):  # make GCS hyperlink to bucket/folder/file
    tmpl = '=HYPERLINK("storage.cloud.google.com/{0}/{1}/{2}", "{2}")'
    return tmpl.format(bucket, folder, fname)

def main(fname, bucket, SHEET_ID, folder):
    fname, mtype, ftime, data = drive_get_img(fname)
    gcs_blob_upload(fname, folder, bucket, data, mtype)
    info = vision_label_img(data)
    sheet_append_row(SHEET_ID, [folder, _linkify(bucket, fname), mtype,
            ftime, _k_ize(data), info])

While this post may feature just pseudocode, a barebones working version can be accomplished with ~80 lines of actual Python. The rest of the code not shown are constants and other auxiliary support. The application gets kicked off with a call to main() passing in a filename, the Cloud Storage bucket to archive it to, a Drive file ID for the Sheet, and a "folder name," e.g., a directory or ZIP archive. Running it several images results in a spreadsheet that looks like this:

Image archive report in Google Sheets

Developers can build this application step-by-step with our "codelab" (free, online, self-paced tutorials) which can be found here. As you journey through this tutorial, its corresponding open source repo features separate folders for each step so you know what state your app should be in after every implemented function. (NOTE: Files are not deleted, so your users have to decide when to their cleanse Drive folders.) For backwards-compatibility, the script is implemented using older Python auth client libraries, but the repo has an "alt" folder featuring alternative versions of the final script that use service accounts, Google Cloud client libraries, and the newer Python auth client libraries.

Finally to save you some clicks, here are links to the API documentation pages for Google Drive, Cloud Storage, Cloud Vision, and Google Sheets. While this sample app deals with a constrained resource issue, we hope it inspires you to consider what's possible with Google developer tools so you can build your own solutions to improve users' lives every day!

Posted by Jon Harmer, Product Manager, Google Workspace

With today being the start of AMP Fest, quite naturally AMP is on our minds. One of the ways that AMP shines is through email. With AMP for Email, brands can change triggered emails from being just another notification to an easy way for a user to always have realtime and relevant context.

Expanding the AMP Ecosystem

We’re excited to be partnering with Verizon Media and Salesforce Marketing Cloud to build for a future in which every message and touchpoint is an opportunity to make a delightful impression with rich, web-like experiences.

“The motivation to join the AMP for email project was simple: Allowing brands to send richer and more engaging emails to our users. This in turn creates a much better user experience. This also enables features and functionality right within the email environment which are on par with other native web or app experiences. It’s a perfect fit with our mission ... to create the best consumer email experience.” said Nirmal Thangaraj, Engineer on Verizon Media Mail, which powers AOL and Yahoo! Mail.

Making things even easier for email senders, Salesforce announced at AMP Fest that early next year, senders will be able to send AMP emails from the Marketing Cloud. With Salesforce Marketing Cloud enabling AMP emails, senders can add one to two actionable steps into their emails and store that information back in Salesforce Marketing Cloud.

AMP for Productivity

Another area where AMP can really make an impact is in the office. With the influx of applications in the workplace, companies are using new SaaS applications to simplify individual processes - but it comes with a downside of complicating a workers day by requiring that employee jump from app to app to get work done. With context aware content that's dynamically populated and updated in real-time, AMP helps make email a place where work gets done .

Let’s take a look at a couple partners who have been building AMP emails, and how they’ve gone about implementing AMP as part of their email strategy.

Guru

Guru sends tens of thousands of notification emails each day, and while helpful, there were limitations to their effectiveness. Here’s Jason Maynard, Guru’s VP of Product on AMP:

“Static emails are helpful for giving a user awareness of a necessary task, but they also require that user to navigate away from their inbox to our web app in order to review knowledge cards and take specific actions. Their workflow is interrupted. Thus, we decided to leverage AMP in hopes of alleviating this user friction with a goal of fostering engagement within an email thread and reducing context switching.”

And the process and results also were in Guru’s favor: “AMP’s predefined components, documented examples, and testing playgrounds were all development resources that enabled us to deploy AMP payloads very quickly.The new implementation has resulted in users now being able to interact with these notifications to a much greater extent. Users can now expand and read knowledge cards within their email thread. They can also complete actions such as card verifications and reply comments. Emails are now much more stateful and relevant to users.”

After deploying AMP, Guru saw a noticeable uptick in email-driven actions resulting in a 2.5x increase in the number of card comment actions and a 75% increase in card verification. These are thousands of new actions that helped teams manage their knowledge base, all without leaving their inbox.

VOGSY

VOGSY, the Professional Services Automation Cloud App for Workspace, sends approval and notification emails that have multiple conversion paths. Historically, these actions would take a day to complete. With AMP, they've seen an 80% improvement in completion speed. Reaching this success was a smooth and pleasant journey.

“Our developers and our users love AMP technology. Developers truly enjoy building engaging emails with personalized content that is securely and dynamically updated every time you open the email. User adoption is 100%. Completing a workflow can be done without leaving your inbox. That is a huge improvement in user experience. Because of its fast adoption, we expect to send more than 2 million AMP emails in the first year,” said Leo Koster, Founder of VOGSY.

Copper

Copper is a CRM designed for people whose business relies on relationship-building, Copper functions seamlessly in the background while employees spend time on what matters: customers. Email is obviously a big part of how organizations communicate, plan, and collaborate. And up to now, email is mostly used as a gateway to other applications where users can take action or complete their task.

“This is why the idea of dynamic emails intrigued us... Supercharging the receivers’ experience to provide up to date information that you can interact with from your inbox. Instead of receiving static email notifications each time you are tagged, we leveraged AMP for email to give users a single, dynamic email where they can see relevant information about the opportunity. They can then respond to comments from their teammates—bringing our users the most seamless experience possible wherever they like to work,” said Sefunmi Osinaike, Product Manager at Copper.

And best of all, the process was simple: “Our developers described the documentation as enjoyable because it helped us add rich components without the overhead of figuring out how to make them work in email with basic HTML. The ease of use of lists, inputs and tooltips accelerated the rate we prototyped our feature and it saved us a lot of time. We also got a ton of support on stack overflow with a response rate in less than 24 hours.”

For Copper, AMP has allowed them to take the experiences that always existed in Copper, but move them closer to the employee’s day-to-day workflow by allowing them to take those actions from email.

Stripo

As an email design platform, Stripo.email has seen over 1,000 different companies create AMP email campaigns with Carousels, Feedback Forms, and Net Promoter Score forms--in one month alone. Stripo was able to implement AMP where users could fill out forms without having to leave their inbox. The strategy drove a 5x lift in effectiveness from traditional questionnaires.

We’re excited about AMP and all of the great use cases partners are implementing to modernize the capabilities of email. To learn more about AMP for Email, click here and be sure to check out AMP Fest.

Posted by Jon Harmer, Product Manager, Google Workspace

With today being the start of AMP Fest, quite naturally AMP is on our minds. One of the ways that AMP shines is through email. With AMP for Email, brands can change triggered emails from being just another notification to an easy way for a user to always have realtime and relevant context.

Expanding the AMP Ecosystem

We’re excited to be partnering with Verizon Media and Salesforce Marketing Cloud to build for a future in which every message and touchpoint is an opportunity to make a delightful impression with rich, web-like experiences.

“The motivation to join the AMP for email project was simple: Allowing brands to send richer and more engaging emails to our users. This in turn creates a much better user experience. This also enables features and functionality right within the email environment which are on par with other native web or app experiences. It’s a perfect fit with our mission ... to create the best consumer email experience.” said Nirmal Thangaraj, Engineer on Verizon Media Mail, which powers AOL and Yahoo! Mail.

Making things even easier for email senders, Salesforce announced at AMP Fest that early next year, senders will be able to send AMP emails from the Marketing Cloud. With Salesforce Marketing Cloud enabling AMP emails, senders can add one to two actionable steps into their emails and store that information back in Salesforce Marketing Cloud.

AMP for Productivity

Another area where AMP can really make an impact is in the office. With the influx of applications in the workplace, companies are using new SaaS applications to simplify individual processes - but it comes with a downside of complicating a workers day by requiring that employee jump from app to app to get work done. With context aware content that's dynamically populated and updated in real-time, AMP helps make email a place where work gets done .

Let’s take a look at a couple partners who have been building AMP emails, and how they’ve gone about implementing AMP as part of their email strategy.

Guru

Guru sends tens of thousands of notification emails each day, and while helpful, there were limitations to their effectiveness. Here’s Jason Maynard, Guru’s VP of Product on AMP:

“Static emails are helpful for giving a user awareness of a necessary task, but they also require that user to navigate away from their inbox to our web app in order to review knowledge cards and take specific actions. Their workflow is interrupted. Thus, we decided to leverage AMP in hopes of alleviating this user friction with a goal of fostering engagement within an email thread and reducing context switching.”

And the process and results also were in Guru’s favor: “AMP’s predefined components, documented examples, and testing playgrounds were all development resources that enabled us to deploy AMP payloads very quickly.The new implementation has resulted in users now being able to interact with these notifications to a much greater extent. Users can now expand and read knowledge cards within their email thread. They can also complete actions such as card verifications and reply comments. Emails are now much more stateful and relevant to users.”

After deploying AMP, Guru saw a noticeable uptick in email-driven actions resulting in a 2.5x increase in the number of card comment actions and a 75% increase in card verification. These are thousands of new actions that helped teams manage their knowledge base, all without leaving their inbox.

VOGSY

VOGSY, the Professional Services Automation Cloud App for Workspace, sends approval and notification emails that have multiple conversion paths. Historically, these actions would take a day to complete. With AMP, they've seen an 80% improvement in completion speed. Reaching this success was a smooth and pleasant journey.

“Our developers and our users love AMP technology. Developers truly enjoy building engaging emails with personalized content that is securely and dynamically updated every time you open the email. User adoption is 100%. Completing a workflow can be done without leaving your inbox. That is a huge improvement in user experience. Because of its fast adoption, we expect to send more than 2 million AMP emails in the first year,” said Leo Koster, Founder of VOGSY.

Copper

Copper is a CRM designed for people whose business relies on relationship-building, Copper functions seamlessly in the background while employees spend time on what matters: customers. Email is obviously a big part of how organizations communicate, plan, and collaborate. And up to now, email is mostly used as a gateway to other applications where users can take action or complete their task.

“This is why the idea of dynamic emails intrigued us... Supercharging the receivers’ experience to provide up to date information that you can interact with from your inbox. Instead of receiving static email notifications each time you are tagged, we leveraged AMP for email to give users a single, dynamic email where they can see relevant information about the opportunity. They can then respond to comments from their teammates—bringing our users the most seamless experience possible wherever they like to work,” said Sefunmi Osinaike, Product Manager at Copper.

And best of all, the process was simple: “Our developers described the documentation as enjoyable because it helped us add rich components without the overhead of figuring out how to make them work in email with basic HTML. The ease of use of lists, inputs and tooltips accelerated the rate we prototyped our feature and it saved us a lot of time. We also got a ton of support on stack overflow with a response rate in less than 24 hours.”

For Copper, AMP has allowed them to take the experiences that always existed in Copper, but move them closer to the employee’s day-to-day workflow by allowing them to take those actions from email.

Stripo

As an email design platform, Stripo.email has seen over 1,000 different companies create AMP email campaigns with Carousels, Feedback Forms, and Net Promoter Score forms--in one month alone. Stripo was able to implement AMP where users could fill out forms without having to leave their inbox. The strategy drove a 5x lift in effectiveness from traditional questionnaires.

We’re excited about AMP and all of the great use cases partners are implementing to modernize the capabilities of email. To learn more about AMP for Email, click here and be sure to check out AMP Fest.