EU member states are in the process of implementing the European Copyright Directive into national law. These changes include the incorporation of Article 15, which introduced a new copyright for press publishers, and Article 17, which introduced new rules for certain content-sharing services like YouTube. Here’s a little more on what YouTube in particular is doing.
Developing our compliance plans
YouTube supports updating copyright rules for the digital age and has been working closely with policymakers across Europe on Article 17’s implementation and on a means of compliance that best meets the needs of our partners’ growing businesses, while also continuing to contribute to the economy. For example, YouTube’s creative ecosystem contributed approximately €775 million to Germany’s GDP in 2019, supporting the equivalent of 25,000 full time equivalent jobs, according to Oxford Economics. As we develop our compliance plans, it’s important to us that the fundamental rights of rightsholders, including media companies, creators, artists and users, are protected. At the same time, supporting the creative industries such as the YouTube creators who are building next generation media companies that impact the economy's overall success, is a priority to us.
Implementing Content ID for better creative control
YouTube has long believed that effective rightsholder management is key to a healthy business and flourishing creativity. This is why we created a set of copyright management tools, for example, investing over 100 million U.S. dollars to develop our Content ID technology which allows rightsholders to manage their content across the platform in a highly nuanced and sophisticated way. YouTube has worked with rightsholders such as record companies, labels, music collecting societies and film and television creators and producers for many years. In fact, Content ID enables rightsholders to control and manage their content on YouTube, while also gaining a new source of income. More than 95% of the time when rightsholders claim content through Content ID, they choose to allow the content to remain on the platform and earn money generated by ads and subscriptions, rather than block the content. This has resulted in an entirely new revenue stream. Over the last three years alone, YouTube paid $30B to media companies, creators and artists, and in the last twelve months, YouTube paid over $4B dollars to the music industry, 30% of which came from user-generated content monetized through Content ID.
Copyright Match Tool
We are also expanding access to more copyright tools for more creators. For example, we recently expanded our Copyright Match Tool so that any rightsholder who successfully submits a copyright removal — whether they host their content on our platform or not — can have access to our matching technology and are able to manage their copyrighted material in a powerful, efficient and user-friendly way. We also allow rightsholders who submit copyright takedown notices to request that YouTube prevent copies of the same content from being uploaded in the future. Additionally, we will be hosting a webform in our help center to enable interested parties to reach out to us for licensing discussions.
We continue to find new ways to help creators and rightsholders manage their rights on YouTube while growing the amounts we pay to the creative industries and will continue to work with policymakers, rights owners, partners and creators across the region to ensure an effective implementation of the new copyright laws.
In March of this year, we announced that beginning September 30, 2021, we will end support for the legacy v2.0 of the Content API for Shopping. We also recommended you migrate to using v2.1, which has been available since March 2019.
As of this announcement, there is now less than 1 month until support for v2.0 will end on September 30, 2021. Following this date, the legacy v2.0 of the Content API is no longer guaranteed to function. We will continue to provide support for your efforts to migrate to v2.1. To avoid disruption we strongly encourage you to migrate to v2.1 imminently.
What do I need to know? To check your usage of Content API for Shopping, you can look it up:
If your application uses a multi-client account (MCA) to make changes to sub-accounts, you should also check your API usage for that account. Please note that all API usage from Content API for Shopping v2.0, is no longer guaranteed to function after September 30, 2021.
If you have any questions or issues during migration, contact your Google representative for Merchant Center programs, or ask questions in the Content API for Shopping forum. Ian Ballantyne, Content API for Shopping Team
Every day, our global teams field questions from publishers seeking to learn more about privacy topics like the end of support for third-party cookies and the Privacy Sandbox.
To help partners stay current on these topics and more, we’ve created a new publisher privacy video series that addresses questions we hear most often. Each episode will be brief – five minutes or less – and will be released every few months to keep partners informed about new developments.
The first episode of the series is designed to provide a base level of understanding around ads privacy before we dive into deeper or more timely topics.
Stay tuned for the second Publisher Privacy Q&A episode coming in October. In episode 2 we’ll take a much closer look at how the Privacy Sandbox protects user privacy and what publishers and advertisers will be able to do with it.
Note - this blog post includes stories of loss, and may be upsetting for some readers.
Google is committed to improving the lives of as many people as possible. This includes promoting awareness to prevent drug-related overdoses and deaths, and highlighting recovery support services for people in or seeking recovery from addiction. For parents like us who’ve lost children to drugs, this mission to help and heal families is deeply personal.
We’ve chosen to share our stories today, which is International Overdose Awareness Day, a global event to bring attention to the issue of drug overdoses and to reduce the stigma associated with both drug-related deaths and non-fatal overdoses. The Centers for Disease Control and Prevention (CDC) reported overdose deaths in the U.S. accelerated during the COVID-19 crisis, as people struggling with depression, anxiety and substance use disorders experienced increased isolation, fear, desperation and hopelessness.
More than 93,000 people died of a drug overdose in the U.S. in 2020 ー the highest number of overdose deaths ever recorded in a 12-month period. As Google employees and fathers who recently lost our own sons to overdoses, these numbers are more than just statistics to us. They are our children’s lives cut short, their futures stolen. We’re sharing our stories here in hopes of helping others gain the knowledge and find the resources to prevent future overdose deaths.
Jan’s story: Remembering Linus
Linus on the beach in Santa Cruz, holding his little brother, Lucas.
Our beloved son Linus passed away on July 22, 2020, when he was only 17-years-old. He was the victim of fentanyl poisoning, along with tens of thousands of other victims in 2020. He had purchased a pill that he thought was Percocet, but it contained a lethal dose of fentanyl. When we discovered Linus, it was too late to revive him. A first-born son, big brother to two younger siblings, a grandchild, a cousin, a wide-hearted, accepting friend, brave and humorous human being, was now gone forever.
Like so many others in the time of COVID-19, Linus struggled with substance abuse issues. Nevertheless, he was a high-functioning teen and a great athlete. He was a charismatic and positive human being with a wide network of friends. His life was tragically taken away when he took that fake pill.
Sadly, what happened to Linus isn’t an isolated case: Around 60% of fatal overdoses in 2020 were caused by fentanyl ー a synthetic opioid that is 100 times stronger than morphine. Fentanyl is mixed into street drugs because it’s cheap and highly addictive. Brennan’s son Aidan also fell victim to this deadly drug.
Brennan’s story: Grateful for Aidan
Aidan in Zion on a family trip.
Tragically, on November 25, 2020, our son Aidan Leaf Mullin passed away from fentanyl poisoning. He had just turned 18, was getting ready to start college in the spring semester and was spending time in Idaho before school began. He was mailed pills he thought were Percocet, but were instead “fentapills” ー counterfeit pills containing fentanyl. He purchased the pills via a social media interaction. Aidan was passionate about the outdoors and playing guitar; he was funny and caring toward his friends and family. Our family is grief-stricken and learning to live with this unexpected loss.
We talked to Aidan about the risks of drugs and were hyper-vigilant at home. The truth is that a loving family, a good education and knowledge about these risks is too often no match against peer pressure and anonymous, friction-free accessibility to dangerous drugs.
Like so many families, neither of ours was fully aware of the prevalence of counterfeit pills. The Drug Enforcement Administration (DEA) warns of an increase in accidental poisoning and overdose deaths related to counterfeit pills containing fentanyl. Just one pill can kill your child, even if they are not using drugs regularly.
Talking to your kids about the dangers of fake pills
Both Jan’s family and mine have been through a period of deep grief, and we now realize how little the general public knows about the deadly dangers of fake pills. Ask your children what they know about this issue. So often kids are simply unaware that these pills are not prescribed by a physician, but manufactured in an illegal lab – most often with fentanyl as their main ingredient.
As a parent or caregiver, it’s worth paying attention to what your child is doing and who they’re communicating with on the internet. Emphasize to your kids that people who buy drugs on the street or online are being duped. Drug dealers only care about profits, not lives.
Resources
Parents and caretakers are faced with the challenge of protecting their loved ones from the risks associated with opioids. Here are some resources to help:
Naloxone (Narcan) can reverse opioid overdoses and save lives. You can use Google Maps’ locator tool on g.co/recovertogether to find Naloxone at pharmacy locations in your community, or you can search by state at Next Naloxone. The American Medical Association (AMA) offers a guide on how to administer Naloxone.
The Substance Abuse and Mental Health Services Administration (SAMHSA) operates a free, confidential, national hotline for people looking for help addressing substance use at: 1-800-662-HELP (4357) and a list of licensed treatment providers at findtreatment.gov.
The Partnership to End Addiction is a long-standing nonprofit helping parents and caregivers find support addressing their children's substance use. They offer a text-based hotline (text CONNECT to 55753), and a free online support community for parents and caregivers.
Google’s Recover Together provides resources for Americans in or seeking recovery from addiction. Search for recovery resources for individuals and families in your community, learn what to expect in the recovery process and hear stories from the voices of those who’ve recovered – and rediscovered hope and purpose in their lives.
If you have opioids or other potentially harmful medicines to dispose of, Google has incorporated drug disposal locations into Google Maps and Search; queries like “medication disposal near me” will display disposal locations.
Note - this blog post includes stories of loss, and may be upsetting for some readers.
Google is committed to improving the lives of as many people as possible. This includes promoting awareness to prevent drug-related overdoses and deaths, and highlighting recovery support services for people in or seeking recovery from addiction. For parents like us who’ve lost children to drugs, this mission to help and heal families is deeply personal.
We’ve chosen to share our stories today, which is International Overdose Awareness Day, a global event to bring attention to the issue of drug overdoses and to reduce the stigma associated with both drug-related deaths and non-fatal overdoses. The Centers for Disease Control and Prevention (CDC) reported overdose deaths in the U.S. accelerated during the COVID-19 crisis, as people struggling with depression, anxiety and substance use disorders experienced increased isolation, fear, desperation and hopelessness.
More than 93,000 people died of a drug overdose in the U.S. in 2020 ー the highest number of overdose deaths ever recorded in a 12-month period. As Google employees and fathers who recently lost our own sons to overdoses, these numbers are more than just statistics to us. They are our children’s lives cut short, their futures stolen. We’re sharing our stories here in hopes of helping others gain the knowledge and find the resources to prevent future overdose deaths.
Jan’s story: Remembering Linus
Linus on the beach in Santa Cruz, holding his little brother, Lucas.
Our beloved son Linus passed away on July 22, 2020, when he was only 17-years-old. He was the victim of fentanyl poisoning, along with tens of thousands of other victims in 2020. He had purchased a pill that he thought was Percocet, but it contained a lethal dose of fentanyl. When we discovered Linus, it was too late to revive him. A first-born son, big brother to two younger siblings, a grandchild, a cousin, a wide-hearted, accepting friend, brave and humorous human being, was now gone forever.
Like so many others in the time of COVID-19, Linus struggled with substance abuse issues. Nevertheless, he was a high-functioning teen and a great athlete. He was a charismatic and positive human being with a wide network of friends. His life was tragically taken away when he took that fake pill.
Sadly, what happened to Linus isn’t an isolated case: Around 60% of fatal overdoses in 2020 were caused by fentanyl ー a synthetic opioid that is 100 times stronger than morphine. Fentanyl is mixed into street drugs because it’s cheap and highly addictive. Brennan’s son Aidan also fell victim to this deadly drug.
Brennan’s story: Grateful for Aidan
Aidan in Zion on a family trip.
Tragically, on November 25, 2020, our son Aidan Leaf Mullin passed away from fentanyl poisoning. He had just turned 18, was getting ready to start college in the spring semester and was spending time in Idaho before school began. He was mailed pills he thought were Percocet, but were instead “fentapills” ー counterfeit pills containing fentanyl. He purchased the pills via a social media interaction. Aidan was passionate about the outdoors and playing guitar; he was funny and caring toward his friends and family. Our family is grief-stricken and learning to live with this unexpected loss.
We talked to Aidan about the risks of drugs and were hyper-vigilant at home. The truth is that a loving family, a good education and knowledge about these risks is too often no match against peer pressure and anonymous, friction-free accessibility to dangerous drugs.
Like so many families, neither of ours was fully aware of the prevalence of counterfeit pills. The Drug Enforcement Administration (DEA) warns of an increase in accidental poisoning and overdose deaths related to counterfeit pills containing fentanyl. Just one pill can kill your child, even if they are not using drugs regularly.
Talking to your kids about the dangers of fake pills
Both Jan’s family and mine have been through a period of deep grief, and we now realize how little the general public knows about the deadly dangers of fake pills. Ask your children what they know about this issue. So often kids are simply unaware that these pills are not prescribed by a physician, but manufactured in an illegal lab – most often with fentanyl as their main ingredient.
As a parent or caregiver, it’s worth paying attention to what your child is doing and who they’re communicating with on the internet. Emphasize to your kids that people who buy drugs on the street or online are being duped. Drug dealers only care about profits, not lives.
Resources
Parents and caretakers are faced with the challenge of protecting their loved ones from the risks associated with opioids. Here are some resources to help:
Naloxone (Narcan) can reverse opioid overdoses and save lives. You can use Google Maps’ locator tool on g.co/recovertogether to find Naloxone at pharmacy locations in your community, or you can search by state at Next Naloxone. The American Medical Association (AMA) offers a guide on how to administer Naloxone.
The Substance Abuse and Mental Health Services Administration (SAMHSA) operates a free, confidential, national hotline for people looking for help addressing substance use at: 1-800-662-HELP (4357) and a list of licensed treatment providers at findtreatment.gov.
The Partnership to End Addiction is a long-standing nonprofit helping parents and caregivers find support addressing their children's substance use. They offer a text-based hotline (text CONNECT to 55753), and a free online support community for parents and caregivers.
Google’s Recover Together provides resources for Americans in or seeking recovery from addiction. Search for recovery resources for individuals and families in your community, learn what to expect in the recovery process and hear stories from the voices of those who’ve recovered – and rediscovered hope and purpose in their lives.
If you have opioids or other potentially harmful medicines to dispose of, Google has incorporated drug disposal locations into Google Maps and Search; queries like “medication disposal near me” will display disposal locations.
Imagine a world where you could decarbonize your business just by asking your energy provider to do so. That’s exactly what we’re setting out to do at Google: to show it can be done, but more importantly, to make it easier for others to do the same.
Today’s agreement with ENGIE, a large European utility, is helping us do just this. As part of our broader announcement that, between now (2021) and 2030, Google will be investing approximately 1 billion euros in digital infrastructure and clean energy in Germany, Google has signed a first-of-its-kind agreement in Europe to purchase the clean energy that will help ensure that our operations in Germany will operate at nearly 80% carbon-free energy on an hourly basis beginning in 2022. In line with our 2030 commitment to operate on carbon-free energy 24/7 at all our campuses and data centers worldwide, we will continue to work to increase this percentage to 100%.
Beyond our own goals, this agreement establishes a roadmap for other companies to follow as we work together to decarbonize electricity use and support Europe’s green recovery.
A new way to buy and sell clean energy
Sourcing carbon-free energy every hour of the day is an incredibly difficult task and will require innovative technological and contractual solutions. In 2010, Google became an early pioneer of purchasing carbon-free energy through individual Power Purchase Agreements (PPAs), complex arrangements that not all energy customers are able to use. Companies can also purchase clean energy via unbundled Energy Attribute Certificates (EACs), which historically have a limited impact on driving clean energy deployment, though we are working to change that. To completely decarbonize our operations -- and to make it easier for other organizations to do the same -- we are working with our partners to rethink how we buy carbon-free electricity.
Under the terms of the agreement we are announcing today, ENGIE will assemble and develop a carbon-free energy portfolio on Google’s behalf that has the ability to flex and grow as our needs change in the region.
Jointly with ENGIE, Google will purchase electricity from 23 renewable energy projects in five German states. Some will be newly-built from scratch. Others (such as existing onshore wind projects that will no longer receive national subsidy support) will see their life extended, so they continue to produce clean electricity instead of being dismantled. By working with our energy suppliers to transform the way clean energy is delivered to customers, Google is supporting Germany's decarbonization goals.
A greener cloud for Germany and beyond
Together with today’s announcements of a new cloud region in Berlin-Brandenburg and the expansion of our existing cloud region in Frankfurt, we are pleased to continue delivering one of the cleanest clouds in the industry to our customers. More and more companies are incorporating environmental, social and governance targetsinto their technology strategies. We’re committed to help IT organizations, and our users, take action today to lower the carbon emissions of their cloud applications. Because of this agreement, European customers will have another low-carbon option where they can run their cloud workloads.
We are excited to add more projects like this in the coming years and continue to work on solutions to reach Google’s ambition to operate on carbon-free energy every hour of every day by 2030, all while supporting Europe’s green transition. Already today, two thirds (67%) of our energy needs are met with locally sourced clean energy on an hourly basis. We will continue to report on our progress, and to share tools and best practices with the wider industry to advance decarbonization on a global scale.
Imagine a world where you could decarbonize your business just by asking your energy provider to do so. That’s exactly what we’re setting out to do at Google: to show it can be done, but more importantly, to make it easier for others to do the same.
Today’s agreement with ENGIE, a large European utility, is helping us do just this. As part of our broader announcement that, between now (2021) and 2030, Google will be investing approximately 1 billion euros in digital infrastructure and clean energy in Germany, Google has signed a first-of-its-kind agreement in Europe to purchase the clean energy that will help ensure that our operations in Germany will operate at nearly 80% carbon-free energy on an hourly basis beginning in 2022. In line with our 2030 commitment to operate on carbon-free energy 24/7 at all our campuses and data centers worldwide, we will continue to work to increase this percentage to 100%.
Beyond our own goals, this agreement establishes a roadmap for other companies to follow as we work together to decarbonize electricity use and support Europe’s green recovery.
A new way to buy and sell clean energy
Sourcing carbon-free energy every hour of the day is an incredibly difficult task and will require innovative technological and contractual solutions. In 2010, Google became an early pioneer of purchasing carbon-free energy through individual Power Purchase Agreements (PPAs), complex arrangements that not all energy customers are able to use. Companies can also purchase clean energy via unbundled Energy Attribute Certificates (EACs), which historically have a limited impact on driving clean energy deployment, though we are working to change that. To completely decarbonize our operations -- and to make it easier for other organizations to do the same -- we are working with our partners to rethink how we buy carbon-free electricity.
Under the terms of the agreement we are announcing today, ENGIE will assemble and develop a carbon-free energy portfolio on Google’s behalf that has the ability to flex and grow as our needs change in the region.
Jointly with ENGIE, Google will purchase electricity from 23 renewable energy projects in five German states. Some will be newly-built from scratch. Others (such as existing onshore wind projects that will no longer receive national subsidy support) will see their life extended, so they continue to produce clean electricity instead of being dismantled. By working with our energy suppliers to transform the way clean energy is delivered to customers, Google is supporting Germany's decarbonization goals.
A greener cloud for Germany and beyond
Together with today’s announcements of a new cloud region in Berlin-Brandenburg and the expansion of our existing cloud region in Frankfurt, we are pleased to continue delivering one of the cleanest clouds in the industry to our customers. More and more companies are incorporating environmental, social and governance targetsinto their technology strategies. We’re committed to help IT organizations, and our users, take action today to lower the carbon emissions of their cloud applications. Because of this agreement, European customers will have another low-carbon option where they can run their cloud workloads.
We are excited to add more projects like this in the coming years and continue to work on solutions to reach Google’s ambition to operate on carbon-free energy every hour of every day by 2030, all while supporting Europe’s green transition. Already today, two thirds (67%) of our energy needs are met with locally sourced clean energy on an hourly basis. We will continue to report on our progress, and to share tools and best practices with the wider industry to advance decarbonization on a global scale.
You can now use Time Insights in Calendar, a personalized, analytical experience on the web, to see how your time is spent across meetings and collaborators.
Time Insights experience for an example calendar week view
Please note, you can only view Time Insights on a computer.
Who’s impacted
Admins and end users
Why you’d use it
With the changes to our working environments in the past year, some people have more meetings and may feel less control over how their work time is spent. Time Insights can show you this data, and help you plan your time better.
Additional details
With Time Insights, you’ll see information such as:
Time breakdown: Based on your working hours and the types of meetings you have
Time in meetings: Highlighting meeting-heavy days and time frames, as well as meeting frequencies
People you meet with: Showing who you spend the most time meeting with. You can also pin key stakeholders to make sure you’re keeping in touch with them. Hovering over an individual will also highlight the meetings on your calendar that include that person
This information is visible to you, not your manager—so you can assess how you’re spending your time against your own priorities.
If you manage other people’s calendars and have “manage sharing access” permission to those calendars, you can view their Time Insights.
Getting started
Admins: This feature will be ON by default and can be disabled at the domain/OU level. Visit the Help Center to learn more about turning Time Insights on or off for your organization. Please note, there is no admin control for Google Workspace Business Standard domains.
End users: There is no end user setting for this feature but users can close the right hand bar out of view at any time. Visit the Help Center to learn more about Time Insights in Calendar.
Rapid Release domains: Gradual rollout (up to 15 days for feature visibility) starting on September 6, 2021
Scheduled Release domains: Gradual rollout (up to 15 days for feature visibility) starting on September 20, 2021
Availability
Available to Google Workspace Business Standard, Business Plus, Enterprise Standard, Enterprise Plus, Education Plus, and Nonprofits customers
Not available to Google Workspace Essentials, Business Starter, Enterprise Essentials, Education Fundamentals, and Frontline, as well as G Suite Basic and Business customers
Hi, everyone! We've just released Chrome 92 (92.0.4515.166) for Android: it'll become available on Google Play over the next few days.
This release includes stability and performance improvements. You can see a full list of the changes in the Git log. If you find a new issue, please let us know by filing a bug.
Posted by Ye Jia, Software Engineer and Julie Cattiau, Product Manager, Google Research
On June 2nd, 2021, Major League Baseball in the United States celebrated Lou Gehrig Day, commemorating both the day in 1925 that Lou Gehrig became the Yankees’ starting first baseman, and the day in 1941 that he passed away from amyotrophic lateral sclerosis (ALS, also known as Lou Gehrig’s disease) at the age of 37. ALS is a progressive neurodegenerative disease that affects motor neurons, which connect the brain with the muscles throughout the body, and govern muscle control and voluntary movements. When voluntary muscle control is affected, people may lose their ability to speak, eat, move and breathe.
In honor of Lou Gehrig, former NFL player and ALS advocate Steve Gleason, who lost his ability to speak due to ALS, recited Gehrig’s famous “Luckiest Man” speech at the June 2nd event using a recreation of his voice generated by a machine learning (ML) model. Gleason’s voice recreation was developed in collaboration with Google’s Project Euphonia, which aims to empower people who have impaired speaking ability due to ALS to better communicate using their own voices.
Steve Gleason, who lost his voice to ALS, worked with Google’s Project Euphonia to generate a speech in his own voice in honor of Lou Gehrig. A portion of Gleason’s speech was broadcast in ballparks across the country during the 4th inning on June 2nd, 2021.
Today we describe PnG NAT, the model adopted by Project Euphonia to recreate Steve Gleason’s voice. PnG NAT is a new text-to-speech synthesis (TTS) model that merges two state-of-the-art technologies, PnG BERT and Non-Attentive Tacotron (NAT), into a single model. It demonstrates significantly better quality and fluency than previous technologies, and represents a promising approach that can be extended to a wider array of users.
Recreating a Voice Non-Attentive Tacotron (NAT) is the successor to Tacotron 2, a sequence-to-sequence neural TTS model proposed in 2017. Tacotron 2 used an attention module to connect the input text sequence and the output speech spectrogram frame sequence, so that the model knows which part of the text to pay attention to when generating each time step of the synthesized speech spectrogram. Tacotron 2 was the first TTS model that was able to synthesize speech that sounds as natural as a person speaking. However, with extensive experimentation we discovered that there is a small probability that the model can suffer from robustness issues — such as babbling, repeating, or skipping part of the text — due to the inherent flexibility of the attention mechanism.
NAT improves upon Tacotron 2 by replacing the attention module with a duration-based upsampler, which predicts a duration for each input phoneme and upsamples the encoded phoneme representation so that the output length corresponds to the length of the predicted speech spectrogram. Such a change both resolves the robustness issue, and improves the naturalness of the synthesized speech. This approach also enables precise control of the speech duration for each phoneme of the input text while still maintaining highly natural synthesis quality. Because recordings of people with ALS often exhibit disfluent speech, this ability to exert per-phoneme control is key for achieving the fluency of the recreated voice.
Non-Attentive Tacotron (NAT) model.
While NAT addresses the robustness issue and enables precise duration control in neural TTS, we build upon it to further improve the natural language understanding of the TTS input. For this, we apply PnG BERT, which uses an approach similar to BERT, but is specifically designed for TTS. It is pre-trained with self-supervision on both the phoneme representation and the grapheme representation of the same content from a large text corpus, and then is used as the encoder of the TTS model. This results in a significant improvement of the prosody and pronunciation of the synthesized speech, especially in difficult cases.
Take, for example, the following audio, which was synthesized from a regular NAT model that takes only phonemes as input:
In comparison, the audio synthesized from PnG NAT on the same input text includes an additional pause that makes the meaning more clear.
The input text to both models is, “To cancel the payment, press one; or to continue, two.” Notice the different pause lengths before the ending “two” in the two versions. The word “two” in the version output by the regular NAT model could be confused for “too”. Because “too” and “two” have identical pronunciation (and thus the same phoneme representation), the regular NAT model does not understand which of the two is appropriate, and assumes it to be the word that more frequently follows a comma, “too”. In contrast, the PnG NAT model can more easily tell the difference, because it takes graphemes in addition to phonemes as input, and thus makes more appropriate pause.
The PnG NAT model integrates the pre-trained PnG BERT model as the encoder to the NAT model. The hidden representations output from the encoder are used by NAT to predict the duration of each phoneme, and are then upsampled to match the length of the audio spectrogram, as outlined above. In the final step, a non-attentive decoder converts the upsampled hidden representations into audio speech spectrograms, which are finally converted into audio waveforms by a neural vocoder.
PnG BERT and the pre-training objectives. Yellow boxes represent phonemes, and pink boxes represent graphemes.
PnG NAT: PnG BERT replaces the original encoder in the NAT model. The random masking for the Masked Language Model (MLM) pre-training is removed.
To recreate Steve Gleason’s voice, we first trained a PnG NAT model with recordings from 31 professional speakers, and then fine-tuned it with 30 minutes of Gleason’s recordings. Because these latter recordings were made after he was diagnosed with ALS, they exhibit signs of slurring. The fine tuned model was able to synthesize speech that sounds very similar to these recordings. However, because the symptoms of ALS were already present in Gleason’s speech, they exhibited some similar disfluencies.
To mitigate this, we leveraged the phoneme duration control of NAT as well as the model trained with professional speakers. We first predicted the durations of each phoneme for both a professional speaker and for Gleason, and then used the geometric mean of the two durations for each phoneme to guide the NAT output. As a result, the model is able to speak in Gleason’s voice, but more fluently than in the original recordings.
Here is the full version of the synthesized Lou Gehrig speech in Gleason’s voice:
Besides recreating voices for people with ALS, PnG NAT is also powering voices for a variety of customers through Google Cloud Custom Voice.
Project Euphonia Of the millions of people around the world who have neurologic conditions that may impact their speech, such as ALS, cerebral palsy or Down syndrome, many may find it difficult to be understood, which can make face-to-face communication challenging. Using voice-activated technologies can be frustrating too, as they don’t always work reliably. Project Euphonia is a Google Research initiative focused on helping people with impaired speech be better understood. The team is researching ways to improve speech recognition for individuals with speech impairments (see recent blog post and segment in TODAY show), as well as customized text-to-speech technology (see Age of AI documentary featuring former NFL player Tim Shaw).
Acknowledgements Many people across Google Research, Google Cloud and Consumer Apps, and Google Accessibility teams contributed to this project and the event, including Michael Brenner, Bob MacDonald, Heiga Zen, Yu Zhang, Jonathan Shen, Isaac Elias, Yonghui Wu, Anne Keck, Danielle Notaro, Kevin Hogan, Zack Kaplan, KR Liu, Kyndra Price, Zoe Ortiz.
Ian Ballantyne, Content API for Shopping Team
