Author Archives: Pandu Nayak

Responsibly applying AI models to Search

For over two decades of Search, we’ve been at the forefront of innovation in language understanding to help deliver on our mission of making the world’s information more accessible and useful for everyone. We’ve seen how critical these advancements are to making information more helpful, and being able to better connect people to creators, publishers and businesses on the web. It’s this constant improvement in understanding human language that’s enabled us to send more traffic to the web every year since Google was created.

We’ve also seen how AI models have significantly improved language innovation. Each successive milestone, from neural nets, to BERT, to MUM, has blown us away with the step changes in information understanding they’ve offered. But with each step forward, we look closely at the limitations and risks new technologies can present.

Across Google, we have been examining the risks and challenges associated with more powerful language models, and we’re committed to responsibly applying AI in Search. Here are some of the ways we do that.

Training on high quality data

We pretrain our models on high-quality data to reduce their potential to perpetuate undesirable biases that may exist in web content. In the case of MUM, we ensured that training data from the web was designated as high-quality based on our search quality metrics, which are informed by our Search Quality Rater Guidelines and driven by our quality rating and evaluation system. This substantially reduces the risk of training on misinformation or explicit content, for example, and is key to our approach.

And as part of our efforts to build a Search experience that works for everyone, MUM was trained on over 75 languages from around the world.

Rigorous Evaluation

Every improvement to Google Search undergoes a rigorous evaluation process to ensure we’re providing more relevant, helpful results. Our Search Quality Rater Guidelines are our north star for how we evaluate great search results. Human raters follow these guidelines and help us understand if our improvements are better fulfilling people’s information needs.

This evaluation process is central to the responsible application of any improvement to Search, whether we’re introducing powerful new systems like BERT or MUM, or simply adding a new feature.

Some changes are bigger than others, so we have to adjust our process accordingly. At the time of its introduction to Search, BERT impacted 1 in 10 English-language queries, so we scaled our evaluation process to be even more rigorous than usual. We subjected our systems to an unprecedented amount of scrutiny, increasing both the scale and granularity of quality testing, to help ensure they weren’t introducing concerning patterns into our systems.

While our standard evaluation process helps us judge launches across a representative query stream, for some improvements, we also more closely examine whether changes provide quality gains or losses across specific slices of queries, or topic areas. This allows us to identify if concerning patterns exist and pursue mitigations before launching an improvement to Search.

Search is not perfect, and any application of AI will not be perfect — this is why any change to Search involves extensive and constant evaluation and testing.

Responsible application design

In addition to working with responsibly designed and trained models, the thoughtful design of products and applications is key to addressing some of the challenges of language models. In Search, many of these critical mitigations take place at the application level, where we can focus on the end-user experience and more effectively manage risk in smaller models designed for specific tasks.

When we adopt new AI technologies such as BERT or MUM, they’re able to help improve individual systems to perform tasks more efficiently and effectively. This approach allows us to focus the scope of our evaluation and understand if an application is introducing concerning patterns. In the event that we do find concerning behavior, we’re able to design much more targeted solutions.

Minding our footprint

Training and running advanced AI models can be energy consumptive. Another benefit of training smaller, application-specific models is that the energy costs of the larger base model, such as MUM, are amortized over the many different applications.

The Google Research team recently published research detailing the energy costs of training state-of-the art language models, and their findings show that combining efficient models, processors, and data centers with clean energy sources can reduce the carbon footprint of a model by as much as one thousand-fold — and we follow this approach to train our models in Search.

Language models in practice

New language models like MUM have enormous potential to transform our ability to understand language and information about the world. And while they may be powerful, they do not make our existing systems obsolete. Today, Google Search employs hundreds of algorithms and machine learning models, none of which are wholly reliant on any singular, large model.

Amongst these hundreds of applications are systems and protections designed specifically to ensure you have a safe, high quality experience. For example, we design our ranking systems to surface relevant and reliable information. Even if a model were to present issues around low quality content, our systems are built to counteract this.

As we’re able to introduce new technologies like MUM into Search, they’ll help us greatly improve our systems and introduce entirely new product experiences. And they can also help us tackle other challenges we face. Improved AI systems can help bolster our spam fighting capabilities and even help us combat known loss patterns. In fact, we recently introduced a BERT-based system to better identify queries seeking explicit content, so we can better avoid shocking or offending users not looking for that information, and ultimately make our Search experience safer for everyone.

We look forward to making Search a better, more helpful product with improved information understanding from these advanced language models, and bringing these new capabilities to Search in a responsible way.

How MUM improved Google Searches for vaccine information

Soda, pop; sweater, jumper; soccer, football. So many things go by different names. Sometimes it’s a function of language, but sometimes it’s a matter of cultural trends or nuance, or simply where you are in the world. 

One very relevant example is COVID-19. As people everywhere searched for information, we had to learn to identify all the different phrases people used to refer to the novel coronavirus to make sure we surfaced high quality and timely information from trusted health authorities like the World Health Organization and Centers for Disease Control and Prevention. A year later, we’re encountering a similar challenge with vaccine names, only this time, we have a new tool to help: Multitask Unified Model (MUM).  


Understanding searches for vaccine information 

AstraZeneca, CoronaVac, Moderna, Pfizer, Sputnik and other broadly distributed vaccines all have many different names all over the world — over 800, based on our analysis. People searching for information about the vaccines may look for “Coronavaccin Pfizer,” “mRNA-1273,” “CoVaccine” — the list goes on. 

Our ability to correctly identify all these names is critical to bringing people the latest trustworthy information about the vaccine. But identifying the different ways people refer to the vaccines all over the world is hugely time-intensive, taking hundreds of human hours. 

With MUM, we were able to identify over 800 variations of vaccine names in more than 50 languages in a matter of seconds. After validating MUM’s findings, we applied them to Google Search so that people could find timely, high-quality information about COVID-19 vaccines worldwide.

Three screenshots of Search results about COVID-19 vaccines.

Surfacing trustworthy information about COVID-19 vaccines in Search.

Transferring knowledge across languages

MUM was able to do a job that should take weeks in just seconds thanks to its knowledge transfer skills. MUM can learn from and transfer knowledge across the 75+ languages it’s trained on. For example, imagine reading a book; if you’re multilingual, you’d be able to share the major takeaways of the book in the other languages you speak — depending on your fluency — because you have an understanding of the book that isn’t language- or translation-dependent. MUM transfers knowledge across languages much like this. 

Similarly, with its knowledge transfer abilities, MUM doesn’t have to learn a new capability or skill in every new language — it can transfer learnings across them, helping us quickly scale improvements even when there isn’t much training data to work with. This is in part thanks to MUM’s sample efficiencies — meaning MUM requires far fewer data inputs than previous models to accomplish the same task. In the case of vaccines, with just a small sample of official vaccine names, MUM was able to rapidly identify these variations across languages.  


Improving Google Search with MUM

This first application of MUM helped us get critical information to users around the world in a timely manner, and we’re looking forward to the many ways in which MUM can make Search more useful to people in the future. Our early testing indicates that not only will MUM be able to improve many aspects of our existing systems, but will also help us create completely new ways to search and explore information.

Improving Search to better protect people from harassment

Over the past two decades of building Google Search, we’ve continued to improve and refine our ability to provide the highest quality results for the billions of queries we see every day. Our core principles guide every improvement, as we constantly update Search to work better for you. One area we’d like to shed more light on is how we balance maximizing access to information with the responsibility to protect people from online harassment.


We design our ranking systems to surface high quality results for as many queries as possible, but some types of queries are more susceptible to bad actors and require specialized solutions. One such example is websites that employ exploitative removals practices. These are sites that require payment to remove content, and since 2018 we’ve had a policy that enables people to request removal of pages with information about them from our results. 


Beyond removing these pages from appearing in Google Search, we also used these removals as a demotion signal in Search, so that sites that have these exploitative practices rank lower in results. This solution leads the industry, and is effective in helping people who are victims of harassment from these sites. 


However, we found that there are some extraordinary cases of repeated harassment. The New York Times highlighted one such case, and shed light on some limitations of our approach.


To help people who are dealing with extraordinary cases of repeated harassment, we’re implementing an improvement to our approach to further protect known victims. Now, once someone has requested a removal from one site with predatory practices, we will automatically apply ranking protections to help prevent content from other similar low quality sites appearing in search results for people’s names. We’re also looking to expand these protections further, as part of our ongoing work in this space.


This change was inspired by a similar approach we’ve taken with victims of non-consensual explicit content, commonly known as revenge porn. While no solution is perfect, our evaluations show that these changes meaningfully improve the quality of our results.


Over the years of building Search, our approach has remained consistent: We take examples of queries where we’re not doing the best job in providing high quality results, and look for ways to make improvements to our algorithms. In this way, we don’t “fix” individual queries, since they’re often a symptom of a class of problems that affect many different queries. Our ability to address issues continues to lead the industry, and we’ve deployed advanced technology, tools and quality signals over the last two decades, making Search work better every day.


Search is never a solved problem, and there are always new challenges we face as the web and the world change. We’re committed to listening to feedback and looking for ways to improve the quality of our results.


Source: Search


MUM: A new AI milestone for understanding information

When I tell people I work on Google Search, I’m sometimes asked, "Is there any work left to be done?" The short answer is an emphatic “Yes!” There are countless challenges we're trying to solve so Google Search works better for you. Today, we’re sharing how we're addressing one many of us can identify with: having to type out many queries and perform many searches to get the answer you need.

Take this scenario: You’ve hiked Mt. Adams. Now you want to hike Mt. Fuji next fall, and you want to know what to do differently to prepare. Today, Google could help you with this, but it would take many thoughtfully considered searches — you’d have to search for the elevation of each mountain, the average temperature in the fall, difficulty of the hiking trails, the right gear to use, and more. After a number of searches, you’d eventually be able to get the answer you need.

But if you were talking to a hiking expert; you could ask one question — “what should I do differently to prepare?” You’d get a thoughtful answer that takes into account the nuances of your task at hand and guides you through the many things to consider.  

This example is not unique — many of us tackle all sorts of tasks that require multiple steps with Google every day. In fact, we find that people issue eight queries on average for complex tasks like this one. 

Today's search engines aren't quite sophisticated enough to answer the way an expert would. But with a new technology called Multitask Unified Model, or MUM, we're getting closer to helping you with these types of complex needs. So in the future, you’ll need fewer searches to get things done. 


Helping you when there isn’t a simple answer

MUM has the potential to transform how Google helps you with complex tasks. Like BERT, MUM is built on a Transformer architecture, but it’s 1,000 times more powerful. MUM not only understands language, but also generates it. It’s trained across 75 different languages and many different tasks at once, allowing it to develop a more comprehensive understanding of information and world knowledge than previous models. And MUM is multimodal, so it understands information across text and images and, in the future, can expand to more modalities like video and audio.

Take the question about hiking Mt. Fuji: MUM could understand you’re comparing two mountains, so elevation and trail information may be relevant. It could also understand that, in the context of hiking, to “prepare” could include things like fitness training as well as finding the right gear. 

Animated GIF visualization representing how MUM interprets the question “I’ve hiked Mt. Adams and now want to hike Mt. Fuji next fall, what should I do to prepare?

Since MUM can surface insights based on its deep knowledge of the world, it could highlight that while both mountains are roughly the same elevation, fall is the rainy season on Mt. Fuji so you might need a waterproof jacket. MUM could also surface helpful subtopics for deeper exploration — like the top-rated gear or best training exercises — with pointers to helpful articles, videos and images from across the web. 


Removing language barriers

Language can be a significant barrier to accessing information. MUM has the potential to break down these boundaries by transferring knowledge across languages. It can learn from sources that aren’t written in the language you wrote your search in, and help bring that information to you. 

Say there’s really helpful information about Mt. Fuji written in Japanese; today, you probably won’t find it if you don’t search in Japanese. But MUM could transfer knowledge from sources across languages, and use those insights to find the most relevant results in your preferred language. So in the future, when you’re searching for information about visiting Mt. Fuji, you might see results like where to enjoy the best views of the mountain, onsen in the area and popular souvenir shops — all information more commonly found when searching in Japanese.

Animated GIF showing a visualization of different illustrations of news sources in different languages.

Understanding information across types

MUM is multimodal, which means it can understand information from different formats like webpages, pictures and more, simultaneously. Eventually, you might be able to take a photo of your hiking boots and ask, “can I use these to hike Mt. Fuji?” MUM would understand the image and connect it with your question to let you know your boots would work just fine. It could then point you to a blog with a list of recommended gear.  

Animated GIF showing a photo of hiking shoes. The question “can I use these to hike Mt. Fuji?” appears next to the shoes.

Applying advanced AI to Search, responsibly

Whenever we take a leap forward with AI to make the world’s information more accessible, we do so responsibly. Every improvement to Google Search undergoes a rigorous evaluation process to ensure we’re providing more relevant, helpful results. Human raters, who follow our Search Quality Rater Guidelines, help us understand how well our results help people find information. 

Just as we’ve carefully tested the many applications of BERT launched since 2019, MUM will undergo the same process as we apply these models in Search. Specifically, we’ll look for patterns that may indicate bias in machine learning to avoid introducing bias into our systems. We’ll also apply learnings from our latest research on how to reduce the carbon footprint of training systems like MUM, to make sure Search keeps running as efficiently as possible.

We’ll bring MUM-powered features and improvements to our products in the coming months and years. Though we’re in the early days of exploring MUM, it’s an important milestone toward a future where Google can understand all of the different ways people naturally communicate and interpret information.

MUM: A new AI milestone for understanding information

When I tell people I work on Google Search, I’m sometimes asked, "Is there any work left to be done?" The short answer is an emphatic “Yes!” There are countless challenges we're trying to solve so Google Search works better for you. Today, we’re sharing how we're addressing one many of us can identify with: having to type out many queries and perform many searches to get the answer you need.

Take this scenario: You’ve hiked Mt. Adams. Now you want to hike Mt. Fuji next fall, and you want to know what to do differently to prepare. Today, Google could help you with this, but it would take many thoughtfully considered searches — you’d have to search for the elevation of each mountain, the average temperature in the fall, difficulty of the hiking trails, the right gear to use, and more. After a number of searches, you’d eventually be able to get the answer you need.

But if you were talking to a hiking expert; you could ask one question — “what should I do differently to prepare?” You’d get a thoughtful answer that takes into account the nuances of your task at hand and guides you through the many things to consider.  

This example is not unique — many of us tackle all sorts of tasks that require multiple steps with Google every day. In fact, we find that people issue eight queries on average for complex tasks like this one. 

Today's search engines aren't quite sophisticated enough to answer the way an expert would. But with a new technology called Multitask Unified Model, or MUM, we're getting closer to helping you with these types of complex needs. So in the future, you’ll need fewer searches to get things done. 


Helping you when there isn’t a simple answer

MUM has the potential to transform how Google helps you with complex tasks. Like BERT, MUM is built on a Transformer architecture, but it’s 1,000 times more powerful. MUM not only understands language, but also generates it. It’s trained across 75 different languages and many different tasks at once, allowing it to develop a more comprehensive understanding of information and world knowledge than previous models. And MUM is multimodal, so it understands information across text and images and, in the future, can expand to more modalities like video and audio.

Take the question about hiking Mt. Fuji: MUM could understand you’re comparing two mountains, so elevation and trail information may be relevant. It could also understand that, in the context of hiking, to “prepare” could include things like fitness training as well as finding the right gear. 

Animated GIF visualization representing how MUM interprets the question “I’ve hiked Mt. Adams and now want to hike Mt. Fuji next fall, what should I do to prepare?

Since MUM can surface insights based on its deep knowledge of the world, it could highlight that while both mountains are roughly the same elevation, fall is the rainy season on Mt. Fuji so you might need a waterproof jacket. MUM could also surface helpful subtopics for deeper exploration — like the top-rated gear or best training exercises — with pointers to helpful articles, videos and images from across the web. 


Removing language barriers

Language can be a significant barrier to accessing information. MUM has the potential to break down these boundaries by transferring knowledge across languages. It can learn from sources that aren’t written in the language you wrote your search in, and help bring that information to you. 

Say there’s really helpful information about Mt. Fuji written in Japanese; today, you probably won’t find it if you don’t search in Japanese. But MUM could transfer knowledge from sources across languages, and use those insights to find the most relevant results in your preferred language. So in the future, when you’re searching for information about visiting Mt. Fuji, you might see results like where to enjoy the best views of the mountain, onsen in the area and popular souvenir shops — all information more commonly found when searching in Japanese.

Animated GIF showing a visualization of different illustrations of news sources in different languages.

Understanding information across types

MUM is multimodal, which means it can understand information from different formats like webpages, pictures and more, simultaneously. Eventually, you might be able to take a photo of your hiking boots and ask, “can I use these to hike Mt. Fuji?” MUM would understand the image and connect it with your question to let you know your boots would work just fine. It could then point you to a blog with a list of recommended gear.  

Animated GIF showing a photo of hiking shoes. The question “can I use these to hike Mt. Fuji?” appears next to the shoes.

Applying advanced AI to Search, responsibly

Whenever we take a leap forward with AI to make the world’s information more accessible, we do so responsibly. Every improvement to Google Search undergoes a rigorous evaluation process to ensure we’re providing more relevant, helpful results. Human raters, who follow our Search Quality Rater Guidelines, help us understand how well our results help people find information. 

Just as we’ve carefully tested the many applications of BERT launched since 2019, MUM will undergo the same process as we apply these models in Search. Specifically, we’ll look for patterns that may indicate bias in machine learning to avoid introducing bias into our systems. We’ll also apply learnings from our latest research on how to reduce the carbon footprint of training systems like MUM, to make sure Search keeps running as efficiently as possible.

We’ll bring MUM-powered features and improvements to our products in the coming months and years. Though we’re in the early days of exploring MUM, it’s an important milestone toward a future where Google can understand all of the different ways people naturally communicate and interpret information.

The ABCs of spelling in Google Search

You’d hardly know it from the way Google Search works, but nearly 20 years after introducing our first spell-check system, spelling remains an ongoing challenge of language understanding. Before we can even begin to start looking for relevant results for a search query, we have to know what a user is looking for, spelled correctly. But every day, one out of 10 search queries is misspelled, and new words are constantly being introduced, along with new ways to misspell them. If you’ve ever been guilty of misspelling a search only to get what you were looking for anyway, read on to learn more about the ABCs of spelling in Google Search. 


A is for All about common spelling mistakes

Our spelling mistakes tend to fall into two main categories: conceptual and slip-of-finger mistakes. We make conceptual mistakes when we’re unsure of how to spell something and try to take our best guess. Say you want to look up the meaning of “gobbledygook” and you don’t know exactly how to spell it, which wouldn’t be unusual since it’s both a difficult to spell word and has two commonly accepted spellings, including “gobbledegook.” In this case, we’ll see many best-effort spellings of the word like “garbledygook,” “gobblydegook,” “gobbleygook,” “gobbly gook,” and more.

An animation of a misspelled search for "gobbledygook"

Slip-of-finger spelling errors happen when we know how to spell what we’re looking for, but accidentally mistype it. Most of us have probably experienced this, especially since the rise of the smartphone, but it happens when we’re typing on full-size keyboards, too. This is why we see over 10,000 variations of queries like “YouTube,” all made by the accidental slip of a finger, such as “ytoube,” “7outub,” “yoitubd” and “tourube.”

An animation of a misspelled search for YouTube

B is for Better models to solve for the unknown

Despite how common our mistakes are, many misspelled queries appear only once, making spelling a unique challenge for Search. And regardless of what kind of spelling error was made, our systems find ways to understand what you mean. Previously, to solve for these never-before-seen misspellings, our systems found inspiration in the keyboard design. For example, if you tried to type "u" but made a mistake, our systems learned you were more likely to have typed "y" than "z" because "y" is adjacent to "u" on a standard English language keyboard. Our models applied the general concept to all new misspellings, walking down nearby letter replacements until a popular replacement term was identified. While this may have seemed like an obvious way to solve for slip-of-finger mistakes, this general approach effectively corrected all kinds of spelling errors, including conceptual mistakes. 

Thanks to advancements in deep learning, we now have a better way to understand spelling. Late last year, we announced a new spelling algorithm that uses a deep neural net that better models and learns from less-common and unique spelling mistakes. This advancement enables us to run a model with more than 680 million parameters in under two milliseconds — a very large model that works faster than the flap of a hummingbird’s wings — so people can search uninterrupted by their own spelling errors. 

And how do our systems know what someone is looking for, no matter the type of mistake and if we’ve never seen the misspelling before? This is where context comes into play. Our natural language understanding models look at a search in context, like the relationship that words and letters within the query have to each other. Our systems start by deciphering or trying to understand your entire search query first. From there, we generate the best replacements for the misspelled words in the query based on our overall understanding of what you’re looking for. For example, we can tell from the other words in the query “average home coast” that you’re probably looking for information on “average home cost.” 


C is for Correcting your query — nicely

You might see these spelling technologies pop up in Google Search in different ways. When we’re pretty sure we know what you’re looking for, we may politely ask, “did you mean…” and show the alternative we think you intended to search for. When we’re very confident that we’ve correctly identified your misspelling, we’ll automatically show results for what we think you’re looking for -- but we’ll always let you know and provide a way to get back to your original spelling. And whether you take our suggestion or not, we’re constantly learning and improving our systems based on that feedback to make Search more helpful.

So whether you’re a spelling bee champ or can't quite nail “I before E except after C,” we'll always be working to improve our spelling so you can keep searching. 


Source: Search


Our latest investments in information quality in Search and News

Delivering high-quality results is what has always set Google apart from other search engines, even in our earliest days. Over the years as the product and user experience have evolved, our investments in quality have accelerated. 


We conduct extensive testing to ensure that Search is as helpful as it can be—from the quality of information we deliver, to the overall experience. Since 2017, we’ve done more than 1 million search quality tests, and we now average more than 1,000 tests per day. 


In addition to investing in the overall Search experience, we also focus on providing reliable information for people everywhere. We’ve highlighted our fundamental approach and ongoing investment in this area, but we also wanted to share some of the new improvements we’ve made to continue to deliver high quality information.


In a year when access to reliable information is more critical than ever—from COVID-19 to natural disasters to important moments of civic participation around the world—our longstanding commitment to quality remains at the core of our mission to make the world’s information accessible and useful. 


New insights from our Intelligence Desk

With new things happening around the world every day, the information landscape can change quickly. To understand how our systems are performing when news breaks, we’ve developed an Intelligence Desk, which actively monitors and identifies potential information threats. 


This effort grew out of our Crisis Response team, which for years has done real-time tracking of events around the world, launching SOS Alerts in Search and Maps to help people get vital information quickly. Over the years, we’ve monitored thousands of events and launched hundreds of alerts to help keep people safe.


intelligence_desk.gif

Crisis events monitored (green) and SOS Alerts launched (red), 2016 - 2020.

The Intelligence Desk is a global team of analysts monitoring news events 24/7, spanning natural disasters and crises, breaking news moments and the latest developments in ongoing topics like COVID. When events occur, our analysts collect data about how our systems are responding and compile reports about narratives that are emerging, like new claims about COVID treatments. Our product teams use these data sets and reports from the Intelligence Desk to run more robust quality tests and ensure that our systems are working as intended for the wide range of topics people Search for.


Improving our systems for breaking news and crises

As news is developing, the freshest information published to the web isn’t always the most accurate or trustworthy, and people’s need for information can accelerate faster than facts can materialize. 


Over the past few years, we’ve improved our systems to automatically recognize breaking news around crisis moments like natural disasters and ensure we’re returning the most authoritative information available. We’ve also made significant strides in our overall ability to accurately identify breaking news moments, and do so more quickly. We’ve improved our detection time from up to 40 minutes just a few years ago, to now within just a few minutes of news breaking.

breaking_news_map.gif

Our improvements in detecting crisis events expands on our work in 2017 to improve the quality of results for topics that might be susceptible to hateful, offensive and misleading information. Those improvements remain fundamental to how we handle low-quality information in Search and News products, and since then, we’ve continuously updated our systems to be able to detect topic areas that may be at risk for misinformation. We’re continuing to train and test our systems to ensure that whatever people are searching for, they can find reliable information.


Providing accurate information from the Knowledge Graph

In Search, features like knowledge panels that display information from the Google Knowledge Graph help you get quick access to the facts from sources across the web. To deliver high-quality information in these features, we’ve deepened our partnerships with government agencies, health organizations and Wikipedia to ensure reliable, accurate information is available, and protect against potential vandalism.


For COVID-19, we worked with health organizations around the world to provide local guidance and information to keep people safe. To respond to emerging information needs, like the surge we saw in people searching for unemployment benefits, we provide easy access to information right from government agencies in the U.S. and other countries. For elections information, we work with non-partisan civic organizations that provide authoritative information about voting methods, candidates, election results and more.


Information in knowledge panels comes from hundreds of sources, and one of the most comprehensive knowledge bases is Wikipedia. Volunteer Wikipedia editors around the world have created robust systems to guard for neutrality and accuracy. They use machine learning tools paired with intricate human oversight to spot and address vandalism. Most vandalism on Wikipedia is reverted within a matter of minutes.
vandalism protection.gif

To complement Wikipedia’s systems, we’ve added additional protections and detection systems to prevent potentially inaccurate information from appearing in knowledge panels. On rare occasions, instances of vandalism on Wikipedia can slip through. Only a small proportion of edits from Wikipedia are potential vandalism, and we’ve improved our systems to now detect 99 percent of those cases. If these issues do appear, we have policies that allow us to take action quickly to address them.


To further support the Wikipedia community, we created the WikiLoop program last year that hosts several editor tools focused on content quality. This includes WikiLoop DoubleCheck, one of a number tools Wikipedia editors and users can use to track changes on a page and flag potential issues. We contribute data from our own detection systems, which members of the community can use to uncover new insights.  


Helpful context from fact checks and Full Coverage

We design Search and News to help you see the full picture, by helping you easily understand the context behind information you might find online. We make it easy to spot fact checks in Search, News and, most recently, Google Images by displaying fact check labels. These fact checks and labels come from publishers that use ClaimReview schemato mark up fact checks they have created. This year to date, people have seen fact checks on Search and News more than 4 billion times, which is more than all of 2019 combined. 


We understand the importance of the fact checking ecosystem in debunking misleading information, which is why we recently donated an additional $6.5 million to help fact checking organizations and nonprofits focus on misinformation about the pandemic.


We also just launched an update using our BERT language understanding models to improve the matching between news stories and available fact checks. These systems can better understand whether a fact check claim is related to the central topic of a story, and surface those fact checks more prominently in Full Coverage—a News feature that provides a complete picture of how a story is reported from a variety of sources. With just a tap, Full Coverage lets you see top headlines from different sources, videos, local news reports, FAQs, social commentary, and a timeline for stories that have played out over time.


Expanded protections for Search features

We have policies for what can appear in Search features like featured snippets, lists or  video previews that uniquely highlight information on the search results page. One notable example is Autocomplete, which helps you complete your search more quickly.


We have long-standing policies to protect against hateful and inappropriate predictions from appearing in Autocomplete. We design our systems to approximate those policies automatically, and have improved our automated systems to not show predictions if we detect that the query may not lead to reliable content. These systems are not perfect or precise, so we enforce our policies if predictions slip through.


We expanded our Autocomplete policies related to elections, and we will remove predictions that could be interpreted as claims for or against any candidate or political party. We will also remove predictions that could be interpreted as a claim about participation in the election—like statements about voting methods, requirements, or the status of voting locations—or the integrity or legitimacy of electoral processes, such as the security of the election. What this means in practice is that predictions like “you can vote by phone” as well as “you can't vote by phone,” or a prediction that says “donate to” any party or candidate, should not appear in Autocomplete. Whether or not a prediction appears, you can still search for whatever you’d like and find results. 


Information online is constantly changing—as are the things people search for—so continuing to deliver high-quality information is an area of ongoing investment. We’ve made great strides and built upon successful improvements to our systems, and we’ll continue to look for new ways to make Search and News as reliable and helpful as possible, no matter what you’re looking for.


Source: Search


Understanding searches better than ever before

If there’s one thing I’ve learned over the 15 years working on Google Search, it’s that people’s curiosity is endless. We see billions of searches every day, and 15 percent of those queries are ones we haven’t seen before--so we’ve built ways to return results for queries we can’t anticipate.

When people like you or I come to Search, we aren’t always quite sure about the best way to formulate a query. We might not know the right words to use, or how to spell something, because often times, we come to Search looking to learn--we don’t necessarily have the knowledge to begin with. 

At its core, Search is about understanding language. It’s our job to figure out what you’re searching for and surface helpful information from the web, no matter how you spell or combine the words in your query. While we’ve continued to improve our language understanding capabilities over the years, we sometimes still don’t quite get it right, particularly with complex or conversational queries. In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question. 

With the latest advancements from our research team in the science of language understanding--made possible by machine learning--we’re making a significant improvement to how we understand queries, representing the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search. 

Applying BERT models to Search
Last year, we introduced and open-sourced a neural network-based technique for natural language processing (NLP) pre-training called Bidirectional Encoder Representations from Transformers, or as we call it--BERT, for short. This technology enables anyone to train their own state-of-the-art question answering system. 

This breakthrough was the result of Google research on transformers: models that process words in relation to all the other words in a sentence, rather than one-by-one in order. BERT models can therefore consider the full context of a word by looking at the words that come before and after it—particularly useful for understanding the intent behind search queries.

But it’s not just advancements in software that can make this possible: we needed new hardware too. Some of the models we can build with BERT are so complex that they push the limits of what we can do using traditional hardware, so for the first time we’re using the latest Cloud TPUsto serve search results and get you more relevant information quickly. 

Cracking your queries
So that’s a lot of technical details, but what does it all mean for you? Well, by applying BERT models to both ranking and featured snippets in Search, we’re able to do a much better job  helping you find useful information. In fact, when it comes to ranking results, BERT will help Search better understand one in 10 searches in the U.S. in English, and we’ll bring this to more languages and locales over time.

Particularly for longer, more conversational queries, or searches where prepositions like “for” and “to” matter a lot to the meaning, Search will be able to understand the context of the words in your query. You can search in a way that feels natural for you.

To launch these improvements, we did a lot of testing to ensure that the changes actually are more helpful. Here are some of the examples that showed up our evaluation process that demonstrate BERT’s ability to understand the intent behind your search.

Here’s a search for “2019 brazil traveler to usa need a visa.” The word “to” and its relationship to the other words in the query are particularly important to understanding the meaning. It’s about a Brazilian traveling to the U.S., and not the other way around. Previously, our algorithms wouldn't understand the importance of this connection, and we returned results about U.S. citizens traveling to Brazil. With BERT, Search is able to grasp this nuance and know that the very common word “to” actually matters a lot here, and we can provide a much more relevant result for this query.

BERT in Search: Visa Example

Let’s look at another query: “do estheticians stand a lot at work.” Previously, our systems were taking an approach of matching keywords, matching the term “stand-alone” in the result with the word “stand” in the query. But that isn’t the right use of the word “stand” in context. Our BERT models, on the other hand, understand that “stand” is related to the concept of the physical demands of a job, and displays a more useful response.

BERT in Search: Esthetician Example

Here are some other examples where BERT has helped us grasp the subtle nuances of language that computers don’t quite understand the way humans do.

Improving Search in more languages
We’re also applying BERT to make Search better for people across the world. A powerful characteristic of these systems is that they can take learnings from one language and apply them to others. So we can take models that learn from improvements in English (a language where the vast majority of web content exists) and apply them to other languages. This helps us better return relevant results in the many languages that Search is offered in.

For featured snippets, we’re using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi and Portuguese.

Search is not a solved problem
No matter what you’re looking for, or what language you speak, we hope you’re able to let go of some of your keyword-ese and search in a way that feels natural for you. But you’ll still stump Google from time to time. Even with BERT, we don’t always get it right. If you search for “what state is south of Nebraska,” BERT’s best guess is a community called “South Nebraska.” (If you've got a feeling it's not in Kansas, you're right.)

Language understanding remains an ongoing challenge, and it keeps us motivated to continue to improve Search. We’re always getting better and working to find the meaning in-- and most helpful information for-- every query you send our way.