Access to information is at the core of Google’s mission, and every day we work to make information from the web available to everyone. We design our systems to return the most relevant and reliable information possible, but our search results include pages from the open web. Depending on what you search for, the results can include content that people might find objectionable or offensive.

While we’re committed to providing open access to information, we also have a strong commitment and responsibility to comply with the law and protect our users. When content is against local law, we remove it from being accessible in Google Search. 

Overall, our approach to information quality and webpage removals aims to strike a balance between ensuring that people have access to the information they need, while also doing our best to protect against harmful information online. Here’s an overview of how we do that.

Complying with the law

We hold ourselves to a high standard when it comes to our legal requirements to remove pages from Google search results. For many issues, such as privacy or defamation, our legal obligations may vary country by country, as different jurisdictions have come to different conclusions about how to deal with these complex topics.

We encourage people and authorities to alert us to content they believe violates the law. In fact, in most cases, this is necessary, because determining whether content is illegal is not always a determination that Google is equipped to make, especially without notice from those who are affected. 

For example, in the case of copyrighted material, we can’t automatically confirm whether a given page hosting that particular content has a license to do so, so we need rightsholders to tell us. By contrast, the mere presence of child sex abuse material (CSAM) on a page is illegal in most jurisdictions, so we develop ways to automatically identify that content and prevent it from showing in our results.

In the case of all legal removals, we share information about government requests for removal in our Transparency Report. Where possible, we inform website owners about requests for removal via Search Console.

Voluntary removal policies

Beyond removing content as required by law, we also have a set of policies that go beyond what’s legally required, mostly focused on highly personal content appearing on the open web. Examples of this content include financial or medical information, government-issued IDs, and intimate imagery published without consent.

These types of content are information that people generally intend to keep private and can cause serious harm, like identity theft, so we give people the ability to request removal from our search results.

We also look for new ways to carefully expand these policies to allow further protections for people online. For example, we allow people to request the removal of pages about themselves on sites with exploitative removals policies, as well as pages that include contact information alongside personal threats, a form of “doxxing.” In these cases, while people may want to access these sites to find potentially useful information or understand their policies and practices, the pages themselves provide little value or public interest, and might lead to reputational or even physical harm that we aim to help protect against.

Solving issues at scale

It might seem intuitive to solve content problems by removing more content — either page by page, or by limiting access to entire sites. However, in addition to being in tension with our mission, this approach also doesn’t effectively scale to the size of the open web, with trillions of pages and more being added each minute. Building scalable, automated approaches allows us to not only solve these challenges more effectively, but also avoid unnecessarily limiting access to legal content online.

Our most effective protection is to design systems that rank high-quality, reliable information at the top of our results. And while we do remove pages in compliance with our policies and legal obligations, we also use insights from those removals to improve our systems overall.

An evolving web

Ultimately, it’s important to remember that even when we remove content from Google Search, it may still exist on the web, and only a website owner can remove content entirely. But we do fight against the harmful effects of sensitive personal information appearing in our results, and have strict practices to ensure we’re complying with the law. We’re always evolving our approach to protect against bad actors on the web and ensure Google continues to deliver high-quality, reliable information for everyone. 

For many people, Google Search is a place they go when they want to find information about a question, whether it’s to learn more about an issue, or fact check a friend quoting a stat about your favorite team. We get billions of queries every day, and one of the reasons people continue to come to Google is they know that they can often find relevant, reliable information that they can trust.

Delivering a high-quality search experience is core to what makes Google so helpful. From the early days when we introduced the PageRank algorithm, understanding the quality of web content was what set Google apart from other search engines.

But people often ask: What do you mean by quality, and how do you figure out how to ensure that the information people find on Google is reliable?

A simple way to think about it is that there are three key elements to our approach to information quality:

• First, we fundamentally design our ranking systems to identify information that people are likely to find useful and reliable. 

• To complement those efforts, we also have developed a number of Search features that not only help you make sense of all the information you’re seeing online, but that also provide direct access to information from authorities—like health organizations or government entities. 

• Finally, we have policies for what can appear in Search features to make sure that we’re showing high quality and helpful content.

With these three approaches, we’re able to continue to improve Search and raise the bar on quality to deliver a trusted experience for people around the world. Let’s take a closer look at how we approach each of these areas.

Orienting our ranking systems around quality 

To understand what results are most relevant to your query, we have a variety of language understanding systems that aim to match the words and concepts in your query with related information in our index. This ranges from systems that understand things like misspellings or synonyms, to more advanced AI-based systems like our BERT-based language capabilities that can understand more complex, natural-language queries. 

Updates to our language understanding systems certainly make Search results more relevant and improve the experience overall. But when it comes to high-quality, trustworthy information, even with our advanced information understanding capabilities, search engines like Google do not understand content the way humans do. We often can’t tell from the words or images alone if something is exaggerated, incorrect, low-quality or otherwise unhelpful.

Instead, search engines largely understand the quality of content through what are commonly called “signals.” You can think of these as clues about the characteristics of a page that align with what humans might interpret as high quality or reliable. For example, the number of quality pages that link to a particular page is a signal that a page may be a trusted source of information on a topic.

We consider a variety of other quality signals, and to understand if our mixture of quality signals is working, we run a lot of tests. We have more than 10,000 search quality raters, people who collectively perform millions of sample searches and rate the quality of the results according to how well they measure up against what we call E-A-T: Expertise, Authoritativeness and Trustworthiness. 

Raters, following instructions anyone can read in our Search Quality Rater Guidelines, evaluate results for sample queries and assess how well the pages listed appear to demonstrate these characteristics of quality.

We recently explained the search rater process in more depth, but it’s worth noting again the ratings we receive are not used directly in our ranking algorithms. Instead, ratings provide data that, when taken in aggregate, help us measure how well our systems are working to deliver quality content that’s aligned with how people—across the country and around the world—evaluate information. This data helps us to improve our systems and ensure we’re delivering high quality results.

For topics where quality information is particularly important—like health, finance, civic information, and crisis situations—we place an even greater emphasis on factors related to expertise and trustworthiness. We’ve learned that sites that demonstrate authoritativeness and expertise on a topic are less likely to publish false or misleading information, so if we can build our systems to identify signals of those characteristics, we can continue to provide reliable information. The design of these systems is our greatest defense against low-quality content, including potential misinformation, and is work that we’ve been investing in for many years.

Info from experts, right in Search

In most cases, our ranking systems do a very good job of making it easy to find relevant and reliable information from the open web, particularly for topics like health, or in times of crisis. But in these areas, we also develop features to make information from authoritative organizations like local governments, health agencies and elections commissions available directly on Search.

For example, we’ve long had knowledge panels in Search with information about health conditions and symptoms, vetted by medical experts. More recently, we saw a significant increase in people searching for information about unemployment benefits, so we worked with administrative agencies to highlight details about eligibility and how to access this civic service. And for many years, we’ve offered features that help you find out how to vote and where your polling place is. Through the Google Civic Information API, we help other sites and services make this information available across the web. This type of information is not always easy to find, especially in rapidly changing situations, so features like these help ensure people get critical guidance when they need it most.

Helping you understand information you see

For many searches, people aren’t necessarily looking for a quick fact, but rather to understand a more complex topic. We also know that people come to Search having heard information elsewhere, with the aim of seeing what others are saying to form their own opinion.

In these cases, we want to give people tools to make sense of the information they’re seeing online, to find reliable sources and explore the full picture about a topic. 

For example, we make it easy to spot fact checks in Search, News, and now in Google Images by displaying fact check labels. These labels come from publishers that use ClaimReview schema to mark up fact checks they have published. For years now we’ve offered Full Coverage on Google News and Search, helping people explore and understand how stories have evolved and explore different angles and perspectives.

Protecting Search features through policies

We also offer more general Search features, like knowledge panels, featured snippets and Autocomplete, that highlight and organize information in unique ways or predict queries you might want to do. Because of the way these features highlight information in Search, we hold ourselves to a very high standard for quality and have guidelines around what content should appear in those spaces.

Within these features, we first and foremost design our automated ranking systems to show helpful content. But our systems aren’t always perfect. So if our systems fail to prevent policy-violating content from appearing, our enforcement team will take action in accordance with our policies. 

To learn more about how we approach policies for our search features, visit this post. And if you’re still looking for more details about Search, check out more past articles in our How Search Works series.

Every Google search you do is one of billions we receive that day. In less than half a second, our systems sort through hundreds of billions of web pages to try and find the most relevant and helpful results available.

Because the web and people’s information needs keep changing, we make a lot of improvements to our search algorithms to keep up. Thousands per year, in fact. And we’re always working on new ways to make our results more helpful whether it’s a new feature, or bringing new language understanding capabilities to Search.

The improvements we make go through an evaluation process designed so that people around the world continue to find Google useful for whatever they’re looking for. Here are some ways that insights and feedback from people around the world help make Search better.

Our research team at work

Changes that we make to Search are aimed at making it easier for people to find useful information, but depending on their interests, what language they speak, and where they are in the world, different people have different information needs. It’s our mission to make information universally accessible and useful, and we are committed to serving all of our users in pursuit of that goal.

This is why we have a research team whose job it is to talk to people all around the world to understand how Search can be more useful. We invite people to give us feedback on different iterations of our projects and we do field research to understand how people in different communities access information online.

For example, we’ve learned over the years about the unique needs and technical limitations that people in emerging markets have when accessing information online. So we developed Google Go, a lightweight search app that works well with less powerful phones and less reliable connections. On Google Go, we’ve also introduced uniquely helpful features, including one that lets you listen to web pages out loud, which is particularly useful for people learning a new language or who may be less comfortable with reading long text. Features like these would not be possible without insights from the people who will ultimately use them.

Search quality raters

A key part of our evaluation process is getting feedback from everyday users about whether our ranking systems and proposed improvements are working well. But what do we mean by “working well”? We publish publicly available rater guidelines that describe in great detail how our systems intend to surface great content. These guidelines are more than 160 pages long, but if we have to boil it down to just a phrase, we like to say that Search is designed to return relevant results from the most reliable sources available.

Our systems use signals from the web itself—like where words in your search appear on web pages, or how pages link to one another on the web—to understand what information is related to your query and whether it’s information that people tend to trust. But notions of relevance and trustworthiness are ultimately human judgments, so to measure whether our systems are in fact understanding these correctly, we need to gather insights from people.

To do this, we have a group of more than 10,000 people all over the world we call “search quality raters.” Raters help us measure how people are likely to experience our results. They provide ratings based on our guidelines and represent real users and their likely information needs, using their best judgment to represent their locale. These people study and are tested on our rater guidelines before they can begin to provide ratings.

How rating works

Here’s how a rater task works: we generate a sample of queries (say, a few hundred). A group of raters will be assigned this set of queries, and they’re shown two versions of results pages for those searches. One set of results is from the current version of Google, and the other set is from an improvement we’re considering.

Raters review every page listed in the results set and evaluate that page against the query, based on our rater guidelines. They evaluate whether those pages meet the information needs based on their understanding of what that query was seeking, and they consider things like how authoritative and trustworthy that source seems to be on the topic in the query. To evaluate things like expertise, authoritativeness, and trustworthiness—sometimes referred to as “E-A-T”—raters are asked to do reputational research on the sources.

Here’s what that looks like in practice: imagine the sample query is “carrot cake recipe.” The results set may include articles from recipe sites, food magazines, food brands and perhaps blogs. To determine if a webpage meets their information needs, a rater might consider how easy the cooking instructions are to understand, how helpful the recipe is in terms of visual instructions and imagery, and whether there are other useful features on the site, like a shopping list creator or calculator for recipe doubling. 

To understand if the author has subject matter expertise, a rater would do some online research to see if the author has cooking credentials, has been profiled or referenced on other food websites, or has produced other great content that has garnered positive reviews or ratings on recipe sites. Basically, they do some digging to answer questions like: is this page trustworthy, and does it come from a site or author with a good reputation?  

Ratings are not used directly for search ranking

Once raters have done this research, they then provide a quality rating for each page. It’s important to note that this rating does not directly impact how this page or site ranks in Search. Nobody is deciding that any given source is “authoritative” or “trustworthy.” In particular, pages are not assigned ratings as a way to determine how well to rank them. Indeed, that would be an impossible task and a poor signal for us to use. With hundreds of billions of pages that are constantly changing, there’s no way humans could evaluate every page on a recurring basis.

Instead, ratings are a data point that, when taken in aggregate, helps us measure how well our systems are working to deliver great content that’s aligned with how people—across the country and around the world—evaluate information.

Last year alone, we did more than 383,605 search quality tests and 62,937 side-by-side experiments with our search quality raters to measure the quality of our results and help us make more than 3,600 improvements to our search algorithms. 

In-product experiments

Our research and rater feedback isn’t the only feedback we use when making improvements. We also need to understand how a new feature will work when it’s actually available in Search and people are using it as they would in real life. To make sure we’re able to get these insights, we test how people interact with new features through live experiments.

They’re called “live” experiments because they’re actually available to a small proportion of randomly selected people using the current version of Search. To test a change, we will launch a feature to a small percentage of all queries we get, and we look at a number of different metrics to measure the impact.

Did people click or tap on the new feature? Did most people just scroll past it? Did it make the page load slower? These insights can help us understand quite a bit about whether a new feature or change is helpful and if people will actually use it.

In 2019, we ran more than 17,000 live traffic experiments to test out new features and improvements to Search. If you compare that with how many launches actually happened (around 3600, remember?), you can see that only the best and most useful improvements make it into Search.

Always improving

While our search results will never be perfect, these research and evaluation processes have proven to be very effective over the past two decades. They allow us to make frequent improvements and ensure that the changes we make represent the needs of people around the world coming to Search for information.

When you come to Search with a query in mind, you trust that Google will find a number of relevant and helpful pages to choose from. We put a lot of time and effort into improving our search systems to ensure that’s the case.

Working on improvements to our language understanding and other search systems is only part of why Google remains so helpful. Equally important is our ability to fight spam. Without our spam-fighting systems and teams, the quality of Search would be reduced--it would be a lot harder to find helpful information you can trust. 

With low quality pages spamming their way into the top results, the greater the chances that people could get tricked by phony sites trying to steal personal information or infect their computers with malware. If you’ve ever gone into your spam folder in Gmail, that’s akin to what Search results would be like without our spam detection capabilities.

Every year we publish a Webspam Report that details the efforts behind reducing spam in your search results and supporting the community of site creators whose websites we help you discover. To coincide with this year’s report, we wanted to give some additional context for why spam-fighting is so important, and how we go about it.

Defining “spam”

We’ve always designed our systems to prioritize the most relevant and reliable webpages at the top. We publicly describe the factors that go into our ranking systems so that web creators can understand the types of content that our systems will recognize as high quality.

We define “spam” as using techniques that attempt to mimic these signals without actually delivering on the promise of a high quality content, or other tactics that might prove harmful to searchers.

Our Webmaster Guidelines detail the types of spammy behavior that is discouraged and can lead to a lower ranking: everything from scraping pages and keyword stuffing to participating in link schemes and implementing sneaky redirects. 

Fighting spam is never-ending battle, a constant game of cat-and-mouse against existing and new spammy behaviors. This threat of spam is why we’ve continued to be very careful about how much detail we reveal about how our systems work. However, we do share a lot, including resources that provide transparency about the positive behaviors creators should follow to create great information and gain visibility and traffic from Search.

Spotting the spammers

The first step of fighting spam is detection. So how do we spot it? We employ a combination of manual reviews by our analysts and a variety of automated detection systems.

We can’t share the specific techniques we use for spam fighting because that would weaken our protections and ultimately make Search much less useful. But we can share about spammy behavior that can be detected systematically. 

After all, a low quality page might include the right words and phrases that match what you searched for, so our language systems wouldn’t be able to detect unhelpful pages from content alone. The telltale signs of spam are in the behavioral tactics used and how they try to manipulate our ranking systems against our Webmaster Guidelines. 

Our spam-fighting systems detect these behaviors so we can tackle this problem at scale. In fact, the scale is huge. Last year, we observed that more than 25 billion of the pages we find each day are spammy. (If each of those pages were a page in a book, that would be more than 20 million copies of “War & Peace” each day!) This leads to an important question: once we find all this spam, what happens next?

Stopping the spammers

When it comes to how we handle spam, it depends on the type of spam and how severe the violation is. For most of the 25 billion spammy pages detected each day, we’re able to automatically recognize their spammy behavior and ensure they don’t rank well in our results. But that’s not the case for everything. 

As with anything, our automated systems aren’t perfect. That’s why we also supplement them with human review, a team that does its own spam sleuthing to understand if content or sites are violating our guidelines. Often, this human review process leads to better automated systems. We look to understand how that spam got past our systems and then work to improve our detection, so that we catch the particular case and automatically detect many other similar cases overall.

In other cases, we may issue what’s called a manual action, when one of our human spam reviewers finds that content that isn’t complying with our Webmaster Guidelines. This can lead to a demotion or a removal of spam content from our search results, especially if it’s deemed to be particularly harmful, like a hacked site that has pages distributing malware to visitors.

When a manual action takes place, we send a notice to the site owner via Search Console, which webmasters can see in their Manual Actions Report. We send millions of these notices each year, and it gives site owners the opportunity to fix the issue and submit for reconsideration. After all, not all “spam” is purposeful, so if a site owner has inadvertently tried tactics that run afoul of our guidelines, or if their site has been compromised by hackers, we want to ensure they can make things right and have their useful information again available to people in Search. This brings us back to why we invest so much effort in fighting spam: so that Search can bring you good, helpful and safe content from sites across the web.

Discovering great information

It’s unfortunate that there’s so much spam, and so much effort that has to be spent fighting it. But that shouldn’t overshadow the fact there are millions upon millions of businesses, publishers and websites with great content for people to discover. We want them to succeed, and we provide tools, support and guidance to help.

We publish our own Search Engine Optimization Starter Guide to provide tips on how to succeed with appropriate techniques in Search. Our Search Relations team conducts virtual office hours, monitors our Webmaster Community forums, and (when possible!) hosts and participates in events around the world to help site creators improve their presence in Search. We provide a variety of support resources, as well as the Search Console toolset to help creators with search.

We’d also encourage anyone to visit our How Google Search Works site, which shares more generally about how our systems work to generate great search results for everyone.