Tag Archives: Pixel

Google’s Next Generation Music Recognition



In 2017 we launched Now Playing on the Pixel 2, using deep neural networks to bring low-power, always-on music recognition to mobile devices. In developing Now Playing, our goal was to create a small, efficient music recognizer which requires a very small fingerprint for each track in the database, allowing music recognition to be run entirely on-device without an internet connection. As it turns out, Now Playing was not only useful for an on-device music recognizer, but also greatly exceeded the accuracy and efficiency of our then-current server-side system, Sound Search, which was built before the widespread use of deep neural networks. Naturally, we wondered if we could bring the same technology that powers Now Playing to the server-side Sound Search, with the goal of making Google’s music recognition capabilities the best in the world.

Recently, we introduced a new version of Sound Search that is powered by some of the same technology used by Now Playing. You can use it through the Google Search app or the Google Assistant on any Android phone. Just start a voice query, and if there’s music playing near you, a “What’s this song?” suggestion will pop up for you to press. Otherwise, you can just ask, “Hey Google, what’s this song?” With this latest version of Sound Search, you’ll get faster, more accurate results than ever before!
Now Playing versus Sound Search
Now Playing miniaturized music recognition technology such that it was small and efficient enough to be run continuously on a mobile device without noticeable battery impact. To do this we developed an entirely new system using convolutional neural networks to turn a few seconds of audio into a unique “fingerprint.” This fingerprint is then compared against an on-device database holding tens of thousands of songs, which is regularly updated to add newly released tracks and remove those that are no longer popular. In contrast, the server-side Sound Search system is very different, having to match against ~1000x as many songs as Now Playing. Making Sound Search both faster and more accurate with a substantially larger musical library presented several unique challenges. But before we go into that, a few details on how Now Playing works.

The Core Matching Process of Now Playing
Now Playing generates the musical “fingerprint” by projecting the musical features of an eight-second portion of audio into a sequence of low-dimensional embedding spaces consisting of seven two-second clips at 1 second intervals, giving a segmentation like this:
Now Playing then searches the on-device song database, which was generated by processing popular music with the same neural network, for similar embedding sequences. The database search uses a two phase algorithm to identify matching songs, where the first phase uses a fast but inaccurate algorithm which searches the whole song database to find a few likely candidates, and the second phase does a detailed analysis of each candidate to work out which song, if any, is the right one.
  • Matching, phase 1: Finding good candidates: For every embedding, Now Playing performs a nearest neighbor search on the on-device database of songs for similar embeddings. The database uses a hybrid of spatial partitioning and vector quantization to efficiently search through millions of embedding vectors. Because the audio buffer is noisy, this search is approximate, and not every embedding will find a nearby match in the database for the correct song. However, over the whole clip, the chances of finding several nearby embeddings for the correct song are very high, so the search is narrowed to a small set of songs which got multiple hits.
  • Matching, phase 2: Final matching: Because the database search used above is approximate, Now Playing may not find song embeddings which are nearby to some embeddings in our query. Therefore, in order to calculate an accurate similarity score, Now Playing retrieves all embeddings for each song in the database which might be relevant to fill in the “gaps”. Then, given the sequence of embeddings from the audio buffer and another sequence of embeddings from a song in the on-device database, Now Playing estimates their similarity pairwise and adds up the estimates to get the final matching score.
It’s critical to the accuracy of Now Playing to use a sequence of embeddings rather than a single embedding. The fingerprinting neural network is not accurate enough to allow identification of a song from a single embedding alone — each embedding will generate a lot of false positive results. However, combining the results from multiple embeddings allows the false positives to be easily removed, as the correct song will be a match to every embedding, while false positive matches will only be close to one or two embeddings from the input audio.

Scaling up Now Playing for the Sound Search server
So far, we’ve gone into some detail of how Now Playing matches songs to an on-device database. The biggest challenge in going from Now Playing, with tens of thousands of songs, to Sound Search, with tens of millions, is that there are a thousand times as many songs which could give a false positive result. To compensate for this without any other changes, we would have to increase the recognition threshold, which would mean needing more audio to get a confirmed match. However, the goal of the new Sound Search server was to be able to match faster, not slower, than Now Playing, so we didn’t want people to wait 10+ seconds for a result.

As Sound Search is a server-side system, it isn’t limited by processing and storage constraints in the same way Now Playing is. Therefore, we made two major changes to how we do fingerprinting, both of which increased accuracy at the expense of server resources:
  • We quadrupled the size of the neural network used, and increased each embedding from 96 to 128 dimensions, which reduces the amount of work the neural network has to do to pack the high-dimensional input audio into a low-dimensional embedding. This is critical in improving the quality of phase two, which is very dependent on the accuracy of the raw neural network output.
  • We doubled the density of our embeddings — it turns out that fingerprinting audio every 0.5s instead of every 1s doesn’t reduce the quality of the individual embeddings very much, and gives us a huge boost by doubling the number of embeddings we can use for the match.
We also decided to weight our index based on song popularity - in effect, for popular songs, we lower the matching threshold, and we raise it for obscure songs. Overall, this means that we can keep adding more (obscure) songs almost indefinitely to our database without slowing our recognition speed too much.

Conclusion
With Now Playing, we originally set out to use machine learning to create a robust audio fingerprint compact enough to run entirely on a phone. It turned out that we had, in fact, created a very good all-round audio fingerprinting system, and the ideas developed there carried over very well to the server-side Sound Search system, even though the challenges of Sound Search are quite different.

We still think there’s room for improvement though — we don’t always match when music is very quiet or in very noisy environments, and we believe we can make the system even faster. We are continuing to work on these challenges with the goal of providing the next generation in music recognition. We hope you’ll try it the next time you want to find out what song is playing! You can put a shortcut on your home screen like this:
Acknowledgements
We would like to thank Micha Riser, Mihajlo Velimirovic, Marvin Ritter, Ruiqi Guo, Sanjiv Kumar, Stephen Wu, Diego Melendo Casado‎, Katia Naliuka, Jason Sanders, Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi and Blaise Aguera y Arcas‎.

Source: Google AI Blog


Fill in the blanks with #teampixel

Sometimes the best photos are the ones that don’t tell the whole story, daring you to fill in the blanks. And when it comes to simple, yet thought-provoking imagery, there’s no one better than #teampixel. 

Consider this week’s mix of minimalist photos—a white fluff dreaming cat dreams, a floating jellyfish heading deeper down under, and a peep of curiosity before descending the stairs. What’s going on in those photos? And what about behind the camera? You decide:

The next time you capture something with your Pixel, tag it with #teampixel and you could see your work featured on @google, @madebygoogle and The Keyword.

Try out Digital Wellbeing to find your own balance with Pixel

We all love our phones—the cameras capture the memories we make, they find us the best route to work each day, and they answer the questions we have throughout the day. But many of us can probably use a little bit of help disconnecting from our devices from time to time so we can focus on the other things in our lives.

Earlier this year, we previewed Digital Wellbeing, a new set of features across our products that aims to help people achieve their desired balance with the technology they use every day. Starting today, all Pixel users can try these features out for themselves as part of an exclusive beta:

  • The new Dashboard helps you understand how you’re spending time on your phone, with a daily overview, a graphic of how frequently you use different apps, how many times you unlock your phone, and how many notifications you receive.
Digital Wellbeing.jpg
  • App Timers let you limit the amount of time you spend using your favorite apps.  

  • Do Not Disturb helps eliminate the anxiety you may feel as notifications pile up. When you’re in a “devices down” meeting or at dinner with friends, Do Not Disturb can be set to keep all visual interruptions from appearing on your screen, including notifications, as well as the sounds.

  • You can activate the Wind Down feature so that at night, as you get close to bedtime, your device goes into Do Not Disturb mode and your screen fades to grayscale to help you disconnect.

android 9 digital wellbeing - wind down

The Digital Wellbeing beta is available for download on Pixel, Pixel XL, Pixel 2 and Pixel 2 XL now, and it will be pushed as an update to all Pixel devices later this year. Get step-by-step instructions on downloading the beta app and try out Digital Wellbeing for yourself.

Pixel keeps getting better, and with Android 9, it’ll also get a number of other AI-backed features, including Adaptive Battery, Adaptive Brightness, and a revamped look and feel. Find out more about Android 9.

Our goal has always been to deliver our newest features to Pixel users as soon as they are ready, and the Digital Wellbeing beta and Android 9 are part of that. So dig in!

#teampixel’s cool inspiration on hot summer days

Have you ever dreamed of the perfect summer vacation? If it includes endless blue skies, colorful cafes, or ancient cobblestone streets, check out the latest round of shots from #teampixel’s favorite summer spots.

When you go on your next adventure, remember to take us with you by tagging #teampixel. You might find yourself featured on The Keyword, @google or @madebygoogle the next time we’re looking for some cool inspiration on these hot summer days.

Insider Attack Resistance

Posted by Shawn Willden, Staff Software Engineer

Our smart devices, such as mobile phones and tablets, contain a wealth of personal information that needs to be kept safe. Google is constantly trying to find new and better ways to protect that valuable information on Android devices. From partnering with external researchers to find and fix vulnerabilities, to adding new features to the Android platform, we work to make each release and new device safer than the last. This post talks about Google's strategy for making the encryption on Google Pixel 2 devices resistant to various levels of attack—from platform, to hardware, all the way to the people who create the signing keys for Pixel devices.

We encrypt all user data on Google Pixel devices and protect the encryption keys in secure hardware. The secure hardware runs highly secure firmware that is responsible for checking the user's password. If the password is entered incorrectly, the firmware refuses to decrypt the device. This firmware also limits the rate at which passwords can be checked, making it harder for attackers to use a brute force attack.

To prevent attackers from replacing our firmware with a malicious version, we apply digital signatures. There are two ways for an attacker to defeat the signature checks and install a malicious replacement for firmware: find and exploit vulnerabilities in the signature-checking process or gain access to the signing key and get their malicious version signed so the device will accept it as a legitimate update. The signature-checking software is tiny, isolated, and vetted with extreme thoroughness. Defeating it is hard. The signing keys, however, must exist somewhere, and there must be people who have access to them.

In the past, device makers have focused on safeguarding these keys by storing the keys in secure locations and severely restricting the number of people who have access to them. That's good, but it leaves those people open to attack by coercion or social engineering. That's risky for the employees personally, and we believe it creates too much risk for user data.

To mitigate these risks, Google Pixel 2 devices implement insider attack resistance in the tamper-resistant hardware security module that guards the encryption keys for user data. This helps prevent an attacker who manages to produce properly signed malicious firmware from installing it on the security module in a lost or stolen device without the user's cooperation. Specifically, it is not possible to upgrade the firmware that checks the user's password unless you present the correct user password. There is a way to "force" an upgrade, for example when a returned device is refurbished for resale, but forcing it wipes the secrets used to decrypt the user's data, effectively destroying it.

The Android security team believes that insider attack resistance is an important element of a complete strategy for protecting user data. The Google Pixel 2 demonstrated that it's possible to protect users even against the most highly-privileged insiders. We recommend that all mobile device makers do the same. For help, device makers working to implement insider attack resistance can reach out to the Android security team through their Google contact.

Acknowledgements: This post was developed in joint collaboration with Paul Crowley, Senior Software Engineer

Insider Attack Resistance

Posted by Shawn Willden, Staff Software Engineer

Our smart devices, such as mobile phones and tablets, contain a wealth of personal information that needs to be kept safe. Google is constantly trying to find new and better ways to protect that valuable information on Android devices. From partnering with external researchers to find and fix vulnerabilities, to adding new features to the Android platform, we work to make each release and new device safer than the last. This post talks about Google's strategy for making the encryption on Google Pixel 2 devices resistant to various levels of attack—from platform, to hardware, all the way to the people who create the signing keys for Pixel devices.

We encrypt all user data on Google Pixel devices and protect the encryption keys in secure hardware. The secure hardware runs highly secure firmware that is responsible for checking the user's password. If the password is entered incorrectly, the firmware refuses to decrypt the device. This firmware also limits the rate at which passwords can be checked, making it harder for attackers to use a brute force attack.

To prevent attackers from replacing our firmware with a malicious version, we apply digital signatures. There are two ways for an attacker to defeat the signature checks and install a malicious replacement for firmware: find and exploit vulnerabilities in the signature-checking process or gain access to the signing key and get their malicious version signed so the device will accept it as a legitimate update. The signature-checking software is tiny, isolated, and vetted with extreme thoroughness. Defeating it is hard. The signing keys, however, must exist somewhere, and there must be people who have access to them.

In the past, device makers have focused on safeguarding these keys by storing the keys in secure locations and severely restricting the number of people who have access to them. That's good, but it leaves those people open to attack by coercion or social engineering. That's risky for the employees personally, and we believe it creates too much risk for user data.

To mitigate these risks, Google Pixel 2 devices implement insider attack resistance in the tamper-resistant hardware security module that guards the encryption keys for user data. This helps prevent an attacker who manages to produce properly signed malicious firmware from installing it on the security module in a lost or stolen device without the user's cooperation. Specifically, it is not possible to upgrade the firmware that checks the user's password unless you present the correct user password. There is a way to "force" an upgrade, for example when a returned device is refurbished for resale, but forcing it wipes the secrets used to decrypt the user's data, effectively destroying it.

The Android security team believes that insider attack resistance is an important element of a complete strategy for protecting user data. The Google Pixel 2 demonstrated that it's possible to protect users even against the most highly-privileged insiders. We recommend that all mobile device makers do the same. For help, device makers working to implement insider attack resistance can reach out to the Android security team through their Google contact.

Acknowledgements: This post was developed in joint collaboration with Paul Crowley, Senior Software Engineer

Three, two, one: New ways to control Google Pixel Buds

Google Pixel Buds let you do a lot with just a quick touch. When you use Pixel Buds with your Pixel or other Android device with the Assistant, simply touch and hold the right earbud to ask for your favorite playlist, make a call, send a message or get walking directions to dinner. And, it allows you to control your audio too—just swipe forward or backward to control volume and tap to play or pause your music.


We’re adding three highly-requested features with the latest update that is beginning to roll out today. It’s as easy as 3, 2, 1.


Triple tap: On and off with touch.Pixel Buds can now be manually turned on or off by triple-tapping on the right earbud.

pixelbuds_3taps_detail.gif

Double tap: Next track.Until now, double tapping let you hear notifications as they arrived on your phone. Now you can set double-tap to skip to the next track. To enable this, go to the Pixel Buds’ settings within the Google Assistant app on your phone and enable double-tap to skip to the next track. You can continue to use a Google Assistant voice command to skip tracks, even if you assign two taps to the “next” track feature.

pixelbuds_2taps_detail.gif

One easy switch: Pairing devices made easy. To switch your Pixel Buds connection between your phone and computer (or any device you’ve previously paired), select your Pixel Buds from the BluetoothTM menu of the desired device. Your Pixel Buds will disconnect from the device you were using and connect to the new one.


These updates are starting to roll out today and will be available to everyone by early next week. Go to g.co/pixelbuds to learn more.

Get outside and smell the flowers with #teampixel

Whether you’re into mountain landscapes or city skyscrapers, Pixel photographers take you out on some incredible adventures this week. Sit back, relax, and let #teampixel transport you to a blue paradise, a reflective lake, and even an underwater swim with jellyfish.

The next time you're on an outdoor adventure, tag your Pixel photos with #teampixel and you might be featured on @google or The Keyword.

#teampixel colored outside the lines with interesting shapes this week

This is shaping up to be a colorful week for #teampixel. We love shapes that grab your attention and guide your eyes when you look at a photograph—and the shapes in this week’s photos are sometimes obvious, like in staircases or sidewalk tiles. Other times they surprise you, like in a spiky Joshua tree or a boat’s wake. Can you find all the shapes in this week’s #teampixel images?

If you’d like to be featured on @google and The Keyword, tag your Pixel photos with #teampixel and you might see yourself next.

“A Good Night” with John Legend and Pixel 2

John Legend just premiered the music video for his new single “A Good Night”—and it was filmed entirely on Pixel 2. We spoke to John about the inspiration behind the video and what it’s like living and communicating in the digital age.


Keyword: Tell us about the inspiration behind the music video for “A Good Night.”

John: The song and the video are about a magical night where you meet someone you immediately connect with and can envision a future together. You're single, dating, probably using all the apps people use now (I haven't dated since the advent of Tinder, etc., but it's a fascinating era in romance), and not finding someone special. But one night—maybe caught up in the music or the drinks or whatever—you meet someone that changes everything. That's a good night! 


The video is set in a club and visually represents the swiping left of the apps by literally swiping dating candidates away. But eventually our couple finds each other. And then, during the bridge, we see things transform into a fantasy future where our couple sees themselves getting married. It turns into a beautiful celebration of love and connection. All DJ'ed and hosted by yours truly!

"A Good Night" music video

Why did you decide to shoot it on Pixel?

We thought it would be cool to shoot it on Pixel because the camera is amazing and we were fascinated by the idea of using a smartphone to film a big budget music video, a video that brings the smartphone dating experience to life.


What’s your favorite moment from the video?

I love the transformation scene in the bridge where you see everything freeze and become surreal before we fast forward to a fantasy wedding.


What surprised you about filming the video with smartphones?

It was really seamless. We're so used to doing everything with expensive cameras when we do these major shoots. But the experience of filming with smartphones felt really natural  and the quality from Pixel was amazing.

A Good Night - Pixel 2

How do you think technology like smartphones can help aspiring artists and creators get noticed?

The technology people can use to create has become more accessible and democratized. And there are so many new ways to inexpensively publish your work. This is true in music, filmmaking and other fields. I love it. It's a boon for creativity and gives so many more people the ability to tell their story and express their vision.  

Who are your favorite new filmmakers?

I'm not sure how you define new. My production company just did a fantastic film called "Monster" with first-time feature filmmaker Anthony Mandler. We've worked together before in music videos and commercials, and I think his debut feature film is excellent.

A Good Night gif

The video plays with the idea of apps and social media and how they affect dating. How does the “age of instalove” change the way you talk to your fans?

I talk a lot about what's timeless about love and human connection. The ways we meet have changed a lot, but I think our emotional needs and desires probably aren't all that different.


We can’t pass up the opportunity to ask about the women in your life. How does Pixel help you stay in touch with Chrissy and Luna and record your moments with them?

I love using Pixel to talk to Chrissy and Luna when I'm on the road. We love Duo for video calling. The Pixel camera takes amazing photos and videos too.  


I'm a big Google fan so it's also nice to have a phone that integrates all Google apps so seamlessly. I use Google Docs, Maps, Drive, Calendar and others all the time. It's great to have a phone made for those apps.

Last question. What are you listening to at the moment?

Kendrick Lamar, Frank Ocean, Miguel, SZA—and LOTS of the “Jesus Christ Superstar” soundtrack in preparation for my big role last Easter Sunday!