Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, Alfred Camera.
In this article, we’d like to give you a short overview of Alfred Camera and our experience of using MediaPipe to transform our moving object feature, and how MediaPipe has helped to get things easier to achieve our goals.
Fig.1 Alfred Camera Logo
Alfred Camera is a smart home app for both Android and iOS devices, with over 15 million downloads worldwide. By downloading the app, users are able to turn their spare phones into security cameras and monitors directly, which allows them to watch their homes, shops, pets anytime. The mission of Alfred Camera is to provide affordable home security so that everyone can find peace of mind in this busy world.
The Alfred Camera team is composed of professionals in various fields, including an engineering team with several machine learning and computer vision experts. Our aim is to integrate AI technology into devices that are accessible to everyone.
Alfred Camera currently has a feature called Moving Object Detection, which continuously uses the device’s camera to monitor a target scene. Once it identifies a moving object in the area, the app will begin recording the video and send notifications to the device owner. The machine learning models for detection are hand-crafted and trained by our team using TensorFlow, and run on TensorFlow Lite with good performance even on mid-tier devices. This is important because the app is leveraging old phones and we'd like the feature to reach as many users as possible.
We had started building our AI features at Alfred Camera since 2017. In order to have a solid foundation to support our AI feature requirements for the coming years, we decided to rebuild our real-time video analysis pipeline. At the beginning of the project, the goals were to create a new pipeline which should be 1) modular enough so we could swap core algorithms easily with minimal changes in other parts of the pipeline, 2) having GPU acceleration designed in place, 3) cross-platform as much as possible so there’s no need to create/maintain separate implementations for different platforms. Based on the goals, we had surveyed several open source projects that had the potential but we ended up using none of them as they either fell short on the features or were not providing the readiness/stabilities that we were looking for.
We started a small team to prototype on those goals first for the Android platform. What came later were some tough challenges way above what we originally anticipated. We ran into several major design changes as some key design basics were overlooked. We needed to implement some utilities to do things that sounded trivial but required significant effort to make it right and fast. Dealing with asynchronous processing also led us into a bunch of timing issues, which took the team quite some effort to address. Not to mention debugging on real devices was extremely inefficient and painful.
Things didn't just stop here. Our product is also on iOS and we had to tackle these challenges once again. Moreover, discrepancies in the behavior between the platform-specific implementations introduced additional issues that we needed to resolve.
Even though we finally managed to get the implementations to the confidence level we wanted, that was not a very pleasant experience and we have never stopped thinking if there is a better option.
Google open sourced MediaPipe project in June 2019 and it immediately caught our attention. We were surprised by how it is perfectly aligned with the previous goals we set, and has functionalities that could not have been developed with the amount of engineering resources we had as a small company.
We immediately decided to start an evaluation project by building a new product feature directly using MediaPipe to see if it could live up to all the promises.
To start the evaluation, we decided to migrate our existing moving object feature to see what exactly MediaPipe can do.
Our current Moving Object Detection pipeline consists of the following main components:
Our current implementation has taken GPU into consideration as much as we can. A series of shaders are created to perform the tasks above and the pipeline is designed to avoid moving pixels between CPU/GPU frequently to eliminate the potential performance hits.
The pipeline involves multiple ML models that are conditionally executed, mixed CPU/GPU processing, etc. All the challenges here make it a perfect showcase for how MediaPipe could help develop a complicated pipeline.
MediaPipe provides a lot of code samples for any developer to bootstrap with. We took the Object Detection on Android sample that comes with the project to start with because of the similarity with the back-end part of our pipeline. It did take us sometimes to fully understand the design concepts of MediaPipe and all the tools associated. But with the complete documentation and the great responsiveness from the MediaPipe team, we got up to speed soon to do most of the things we wanted.
That being said, there were a few challenges we needed to overcome on the road to full migration. Our original pipeline of Moving Object Detection takes the input frame asynchronously, but MediaPipe has timestamp bound limitations such that we cannot just show the result in an allochronic way. Meanwhile, we need to gather data through JNI in a specific data format. We came up with a workaround that conquered all the issues under the circumstances, which will be mentioned later.
After wrapping our models and the processing logics into calculators and wired them up, we have successfully transformed our existing implementation and created our first MediaPipe Moving Object Detection pipeline like the figure below, running on Android devices:
Fig.2 Moving Object Detection Graph
We do not block the video frame in the main calculation loop, and set the detection result as an input stream to show the annotation on the screen. The whole graph is designed as a multi-functioned process, the left chunk is the debug annotation and video frame output module, and the rest of the calculation occurs in the rest of the graph, e.g., low light detection, motion triggered detection, cropping of the area of interest and the detection process. In this way, the graph process will naturally separate into real-time display and asynchronous calculation.
As a result, we are able to complete a full processing for detection in under 40ms on a device with Snapdragon 660 chipset. MediaPipe’s tight integration with TensorFlow Lite provides us the flexibility to get even more performance gain by leveraging whatever acceleration techniques available (GPU or DSP) on the device.
The following figure shows the current implementation working in action:
Fig.3 Moving Object Detection running in Alfred Camera
After getting things to run on Android, Desktop GPU (OpenGL-ES) emulation was our next target to evaluate. We are already using OpenGL-ES shaders for some computer vision operations in our pipeline. Having the capability to develop the algorithm on desktop, seeing it work in action before deployment onto mobile platforms is a huge benefit to us. The feature was not ready at the time when the project was first released, but MediaPipe team had soon added Desktop GPU emulation support for Linux in follow-up releases to make this possible. We have used the capability to detect and fix some issues in the graphs we created even before we put things on the mobile devices. Although it currently only works on Linux, it is still a big leap forward for us.
Testing the algorithms and making sure they behave as expected is also a challenge for a camera application. MediaPipe helps us simplify this by using pre-recorded MP4 files as input so we could verify the behavior simply by replaying the files. There is also built-in profiling support that makes it easy for us to locate potential performance bottlenecks.
The result of the evaluation and the feedback from our engineering team were very positive and promising:
With just a few weeks working with MediaPipe, it has shown strong capabilities to fundamentally transform how we develop our products. Without MediaPipe we could have spent months creating the same features without the same level of performance.
Alfred Camera is designed to bring home security with AI to everyone, and MediaPipe has significantly made achieving that goal easier for our team. From Moving Object Detection to future AI-powered features, we are focusing on transforming a basic security camera use case into a smart housekeeper that can help provide even more context that our users care about. With the support of MediaPipe, we have been able to accelerate our development process and bring the features to the market at an unprecedented speed. Our team is really excited about how MediaPipe could help us progress and discover new possibilities, and is looking forward to the enhancements that are yet to come to the project.
Porsche Taylor’s first time riding a motorcycle alone could have gone better. “That first ride, I had absolutely nothing on right: My helmet was too big, I didn’t own a jacket. I might have had on some baseball gloves; everything was just totally upside-down wrong,” she says. “But I wasn’t afraid, it was exhilarating. It was trying something new, being in control. It was that initial feeling of the freedom of the wind.”
Porsche was one of the participants in the Women Riders World Relay, a relay ride that spanned the globe, beginning in February 2019 in Scotland and ending February 2020 in London. WRWR organizers used Google products like Maps, Sheets and Translate to make sure riders not only had constant, up-to-date access to their routes, but also were able to explore and connect with one another along the way.
“The whole team did phenomenally with the amount of time they had to put together the route and figure out the baton passes,” says Porsche. Google Maps was particularly useful for creating Porsche’s route. She and her fellow riders rode from Sept. 25 to Oct. 14, starting in Maine and heading west across the Canadian border, then down through the Southwest to the Mexican border in Texas. They crossed the country, occasionally riding through snowstorms and dropping temperatures. “When you consider the seasons we were riding through, it was a definite challenge for organizers to find routes that weren’t closed down.”
While Google Maps could help the riders along their journey, it couldn’t do anything about inclement weather. “I quit about four times,” laughs Porsche. “Riding in the cold is not my favorite thing to do. But it was a positive experience all the way around; I don’t know that I would ride in the freezing cold again, but I would do a ride with those women again for sure. I always say the bonds are built on the ground: You’re going to love the folks you ride with to death or you won’t be so cool, and I’m happy to say I love those ladies to death.”
Porsche is vocal about the need for more representation for women in the motor sports community, and she says that things like social media visibility and technical tools like Google Hangouts have helped women who may have felt alone in their shared passion find each other. This idea is in part what inspired her to found Black Girls Ride, a magazine and community originally launched as a place for women of color who ride, which has since grown to include all women. What inspired her to launch Black Girls Ride was the lack of representation she saw when she first started riding—especially in long-distance riding. Traditionally, women filled support roles during these cross-country expeditions, taking a literal backseat to men. In fact, Porsche’s first experience on a bike was sitting behind a man, on the back of her cousin’s bike. “I didn’t so much like the feeling of being a passenger...but I loved the feeling of riding.”
Thanks to women like Porsche and the WRWR riders, the world of motor sports is changing. “Women have become fearless and bold enough to take long distance biking trips on their own. We’re witnessing the explosion of the all-female long distance ride, where women take it upon themselves to create rides that cater to them instead of being a subset of an all-male ride. It’s where we get to take our power back.”
Talking about these rides and seeing women taking them via social media and internet communities are crucial, says Porsche, who also mentions using Google Hangouts to connect with riders across the country. “You’re able to see the growth of female riders; women taking these long distance trips and riding solo have always been there—there are women riding today who have been doing this since the 60s—but social media is now shining a light on them.”
That increased visibility is part of Porsche's work with Black Girls Ride. “I knew from riding in LA that there were more of us than the community would admit to. There was no representation in mainstream media, even for women who were riding professionally, there was very little to nothing,” Porsche says. Now "women all over the world are connecting to the Black Girls Ride brand. We have readers in London, Nigeria, France, just about every country you can name. I’m motivated by these women.” Black Girls Ride has become more than a publication, hosting trainings, workshops and events. And while both men and women are included, it’s Porsche’s focus to make sure women riders are invited to the table and that they are given the same representation, advertising and sponsorship opportunities.
Most of all, she just wants women to feel welcome in this world. “It’s always been my goal to create safe spaces for women to ask questions and get the help they need without fear of ridicule,” she says. “And I’m glad I can be a part of creating that.”
Learn more about the women behind WRWR and how they planned their relay at goo.gle/womenriders.
Porsche Taylor’s first time riding a motorcycle alone could have gone better. “That first ride, I had absolutely nothing on right: My helmet was too big, I didn’t own a jacket. I might have had on some baseball gloves; everything was just totally upside-down wrong,” she says. “But I wasn’t afraid, it was exhilarating. It was trying something new, being in control. It was that initial feeling of the freedom of the wind.”
Porsche was one of the participants in the Women Riders World Relay, a relay ride that spanned the globe, beginning in February 2019 in Scotland and ending February 2020 in London. WRWR organizers used Google products like Maps, Sheets and Translate to make sure riders not only had constant, up-to-date access to their routes, but also were able to explore and connect with one another along the way.
“The whole team did phenomenally with the amount of time they had to put together the route and figure out the baton passes,” says Porsche. Google Maps was particularly useful for creating Porsche’s route. She and her fellow riders rode from Sept. 25 to Oct. 14, starting in Maine and heading west across the Canadian border, then down through the Southwest to the Mexican border in Texas. They crossed the country, occasionally riding through snowstorms and dropping temperatures. “When you consider the seasons we were riding through, it was a definite challenge for organizers to find routes that weren’t closed down.”
While Google Maps could help the riders along their journey, it couldn’t do anything about inclement weather. “I quit about four times,” laughs Porsche. “Riding in the cold is not my favorite thing to do. But it was a positive experience all the way around; I don’t know that I would ride in the freezing cold again, but I would do a ride with those women again for sure. I always say the bonds are built on the ground: You’re going to love the folks you ride with to death or you won’t be so cool, and I’m happy to say I love those ladies to death.”
Porsche is vocal about the need for more representation for women in the motor sports community, and she says that things like social media visibility and technical tools like Google Hangouts have helped women who may have felt alone in their shared passion find each other. This idea is in part what inspired her to found Black Girls Ride, a magazine and community originally launched as a place for women of color who ride, which has since grown to include all women. What inspired her to launch Black Girls Ride was the lack of representation she saw when she first started riding—especially in long-distance riding. Traditionally, women filled support roles during these cross-country expeditions, taking a literal backseat to men. In fact, Porsche’s first experience on a bike was sitting behind a man, on the back of her cousin’s bike. “I didn’t so much like the feeling of being a passenger...but I loved the feeling of riding.”
Thanks to women like Porsche and the WRWR riders, the world of motor sports is changing. “Women have become fearless and bold enough to take long distance biking trips on their own. We’re witnessing the explosion of the all-female long distance ride, where women take it upon themselves to create rides that cater to them instead of being a subset of an all-male ride. It’s where we get to take our power back.”
Talking about these rides and seeing women taking them via social media and internet communities are crucial, says Porsche, who also mentions using Google Hangouts to connect with riders across the country. “You’re able to see the growth of female riders; women taking these long distance trips and riding solo have always been there—there are women riding today who have been doing this since the 60s—but social media is now shining a light on them.”
That increased visibility is part of Porsche's work with Black Girls Ride. “I knew from riding in LA that there were more of us than the community would admit to. There was no representation in mainstream media, even for women who were riding professionally, there was very little to nothing,” Porsche says. Now "women all over the world are connecting to the Black Girls Ride brand. We have readers in London, Nigeria, France, just about every country you can name. I’m motivated by these women.” Black Girls Ride has become more than a publication, hosting trainings, workshops and events. And while both men and women are included, it’s Porsche’s focus to make sure women riders are invited to the table and that they are given the same representation, advertising and sponsorship opportunities.
Most of all, she just wants women to feel welcome in this world. “It’s always been my goal to create safe spaces for women to ask questions and get the help they need without fear of ridicule,” she says. “And I’m glad I can be a part of creating that.”
Learn more about the women behind WRWR and how they planned their relay at goo.gle/womenriders.
As the spread of COVID-19 continues, we’re seeing more searches for travel-related information—like travel advisories and trip cancellation policies—so we’re making some changes this week to help you find this information faster.
Stay up to date on travel advisories
When you search on Google for information like flights, hotels, or things to do you’ll start seeing COVID-19 related travel advisories or restrictions for your destination, with links to relevant information from your country’s travel authority when possible. This information will appear on the search results page, at the top of google.com/travel, and in Google Maps when you search for hotels.
Understanding airline policies for flight changes and cancellations
In response to COVID-19, many airlines have adjusted change fee and cancellation policies. When you search on Google for flights with a specific airline or go to Google Flights, we’ll direct you to our Help Center article with more information on airline policies. These policies may change, so be sure to click through to the airline’s website for the latest information.
We hope these resources help you stay safe and make the best decisions about travel plans. We'll continue to work on ways to help as the COVID-19 situation evolves.
The world of book publishing today is, in many ways, dignified and highbrow. But it was a different story in the 16th century, about a hundred years after the invention of the printing press. Publishing was a high-risk, high-reward proposition: With the right backing and enough capital investment, an entrepreneur could become wildly successful. But publishing the wrong thing in the wrong place could be disastrous—even fatal, with governments and religious authorities taking a very severe view of what content was fit to print.
No one knew this better than Christophe Plantin, who set up a publishing house in Antwerp, Belgium, in the mid-16th century. Facing religious intolerance and escaping persecution, he helped put the city on the map as a publishing powerhouse. His own printing operation continued in his family for generations.
Today, Plantin’s home and business are preserved as the Plantin-Moretus Museum, a UNESCO World Heritage Site and home to 25,000 early printed books. Visitors to Antwerp can walk through the rooms where the family lived and worked, and researchers can delve into the collection’s manuscripts, books, archives and original prints.
And now, thanks to a partnership between the City of Antwerp and Google, we will digitize more than 32,000 books from the museum, along with an additional 60,000 books held by the city’s Hendrik Conscience Heritage Library.
In total, more than 100,000 international works published from the sixteenth to the nineteenth century will be made freely accessible in the coming years via Google Books and the library catalogues of both institutions. The scanned volumes, which are no longer subject to copyright, will be full-text searchable, meaning that researchers—as well as members of the public—will be able to search them easily and quickly.
The digitization will start in early 2021, allowing time for both Google and the city to set up the project and establish logistics processes. We expect it will take at least three years, partly because we don’t want to cause too much disruption to library visitors who come to view the materials. The books will be transported securely in batches from Antwerp to our European digitization center. Shortly after each work is scanned, the digital copy will appear on books.google.com, and the libraries will also receive digital copies of their respective works to incorporate into their own catalogues.
Google Books was launched 15 years ago, with the goal of making all books from around the world digitally available and searchable for everyone. This collaboration with the City of Antwerp adds an incredibly rich collection from the Dutch-speaking world to our collection, and brings us a crucial step closer to achieving our mission.
The world of book publishing today is, in many ways, dignified and highbrow. But it was a different story in the 16th century, about a hundred years after the invention of the printing press. Publishing was a high-risk, high-reward proposition: With the right backing and enough capital investment, an entrepreneur could become wildly successful. But publishing the wrong thing in the wrong place could be disastrous—even fatal, with governments and religious authorities taking a very severe view of what content was fit to print.
No one knew this better than Christophe Plantin, who set up a publishing house in Antwerp, Belgium, in the mid-16th century. Facing religious intolerance and escaping persecution, he helped put the city on the map as a publishing powerhouse. His own printing operation continued in his family for generations.
Today, Plantin’s home and business are preserved as the Plantin-Moretus Museum, a UNESCO World Heritage Site and home to 25,000 early printed books. Visitors to Antwerp can walk through the rooms where the family lived and worked, and researchers can delve into the collection’s manuscripts, books, archives and original prints.
And now, thanks to a partnership between the City of Antwerp and Google, we will digitize more than 32,000 books from the museum, along with an additional 60,000 books held by the city’s Hendrik Conscience Heritage Library.
In total, more than 100,000 international works published from the sixteenth to the nineteenth century will be made freely accessible in the coming years via Google Books and the library catalogues of both institutions. The scanned volumes, which are no longer subject to copyright, will be full-text searchable, meaning that researchers—as well as members of the public—will be able to search them easily and quickly.
The digitization will start in early 2021, allowing time for both Google and the city to set up the project and establish logistics processes. We expect it will take at least three years, partly because we don’t want to cause too much disruption to library visitors who come to view the materials. The books will be transported securely in batches from Antwerp to our European digitization center. Shortly after each work is scanned, the digital copy will appear on books.google.com, and the libraries will also receive digital copies of their respective works to incorporate into their own catalogues.
Google Books was launched 15 years ago, with the goal of making all books from around the world digitally available and searchable for everyone. This collaboration with the City of Antwerp adds an incredibly rich collection from the Dutch-speaking world to our collection, and brings us a crucial step closer to achieving our mission.
 |
| Example architecture for an earlier generation RL agent, IMPALA. Inference is done on the actors, usually using inefficient CPUs. Updated model parameters are frequently sent from the learner to the actors increasing bandwidth requirements. |
 |
| Overview of the architecture of SEED RL. In contrast to the IMPALA architecture, the actors only take actions in environments. Inference is executed centrally by the learner on accelerators using batches of data from multiple actors. |
 |
| Frames per second comparing IMPALA and various configurations of SEED RL on DeepMind Lab. SEED RL achieves 2.4M frames per second using 4,160 CPUs. Assuming the same speed, IMPALA would need 14,000 CPUs. |
 |
| Episode return (i.e., the sum of rewards) over time on the DeepMind Lab game “explore_goal_locations_small” using IMPALA and SEED RL. With SEED RL, the time to train is significantly reduced. |
 |
| The score of different architectures on the Google Research Football “Hard” task. We show that by using an input resolution and a larger model, the score is improved, and with more training, the model can significantly outperform the builtin AI. |
