Every Google search you do is one of billions we receive that day. In less than half a second, our systems sort through hundreds of billions of web pages to try and find the most relevant and helpful results available.

Because the web and people’s information needs keep changing, we make a lot of improvements to our search algorithms to keep up. Thousands per year, in fact. And we’re always working on new ways to make our results more helpful whether it’s a new feature, or bringing new language understanding capabilities to Search.

The improvements we make go through an evaluation process designed so that people around the world continue to find Google useful for whatever they’re looking for. Here are some ways that insights and feedback from people around the world help make Search better.

Our research team at work

Changes that we make to Search are aimed at making it easier for people to find useful information, but depending on their interests, what language they speak, and where they are in the world, different people have different information needs. It’s our mission to make information universally accessible and useful, and we are committed to serving all of our users in pursuit of that goal.

This is why we have a research team whose job it is to talk to people all around the world to understand how Search can be more useful. We invite people to give us feedback on different iterations of our projects and we do field research to understand how people in different communities access information online.

For example, we’ve learned over the years about the unique needs and technical limitations that people in emerging markets have when accessing information online. So we developed Google Go, a lightweight search app that works well with less powerful phones and less reliable connections. On Google Go, we’ve also introduced uniquely helpful features, including one that lets you listen to web pages out loud, which is particularly useful for people learning a new language or who may be less comfortable with reading long text. Features like these would not be possible without insights from the people who will ultimately use them.

Search quality raters

A key part of our evaluation process is getting feedback from everyday users about whether our ranking systems and proposed improvements are working well. But what do we mean by “working well”? We publish publicly available rater guidelines that describe in great detail how our systems intend to surface great content. These guidelines are more than 160 pages long, but if we have to boil it down to just a phrase, we like to say that Search is designed to return relevant results from the most reliable sources available.

Our systems use signals from the web itself—like where words in your search appear on web pages, or how pages link to one another on the web—to understand what information is related to your query and whether it’s information that people tend to trust. But notions of relevance and trustworthiness are ultimately human judgments, so to measure whether our systems are in fact understanding these correctly, we need to gather insights from people.

To do this, we have a group of more than 10,000 people all over the world we call “search quality raters.” Raters help us measure how people are likely to experience our results. They provide ratings based on our guidelines and represent real users and their likely information needs, using their best judgment to represent their locale. These people study and are tested on our rater guidelines before they can begin to provide ratings.

How rating works

Here’s how a rater task works: we generate a sample of queries (say, a few hundred). A group of raters will be assigned this set of queries, and they’re shown two versions of results pages for those searches. One set of results is from the current version of Google, and the other set is from an improvement we’re considering.

Raters review every page listed in the results set and evaluate that page against the query, based on our rater guidelines. They evaluate whether those pages meet the information needs based on their understanding of what that query was seeking, and they consider things like how authoritative and trustworthy that source seems to be on the topic in the query. To evaluate things like expertise, authoritativeness, and trustworthiness—sometimes referred to as “E-A-T”—raters are asked to do reputational research on the sources.

Here’s what that looks like in practice: imagine the sample query is “carrot cake recipe.” The results set may include articles from recipe sites, food magazines, food brands and perhaps blogs. To determine if a webpage meets their information needs, a rater might consider how easy the cooking instructions are to understand, how helpful the recipe is in terms of visual instructions and imagery, and whether there are other useful features on the site, like a shopping list creator or calculator for recipe doubling. 

To understand if the author has subject matter expertise, a rater would do some online research to see if the author has cooking credentials, has been profiled or referenced on other food websites, or has produced other great content that has garnered positive reviews or ratings on recipe sites. Basically, they do some digging to answer questions like: is this page trustworthy, and does it come from a site or author with a good reputation?  

Ratings are not used directly for search ranking

Once raters have done this research, they then provide a quality rating for each page. It’s important to note that this rating does not directly impact how this page or site ranks in Search. Nobody is deciding that any given source is “authoritative” or “trustworthy.” In particular, pages are not assigned ratings as a way to determine how well to rank them. Indeed, that would be an impossible task and a poor signal for us to use. With hundreds of billions of pages that are constantly changing, there’s no way humans could evaluate every page on a recurring basis.

Instead, ratings are a data point that, when taken in aggregate, helps us measure how well our systems are working to deliver great content that’s aligned with how people—across the country and around the world—evaluate information.

Last year alone, we did more than 383,605 search quality tests and 62,937 side-by-side experiments with our search quality raters to measure the quality of our results and help us make more than 3,600 improvements to our search algorithms. 

In-product experiments

Our research and rater feedback isn’t the only feedback we use when making improvements. We also need to understand how a new feature will work when it’s actually available in Search and people are using it as they would in real life. To make sure we’re able to get these insights, we test how people interact with new features through live experiments.

They’re called “live” experiments because they’re actually available to a small proportion of randomly selected people using the current version of Search. To test a change, we will launch a feature to a small percentage of all queries we get, and we look at a number of different metrics to measure the impact.

Did people click or tap on the new feature? Did most people just scroll past it? Did it make the page load slower? These insights can help us understand quite a bit about whether a new feature or change is helpful and if people will actually use it.

In 2019, we ran more than 17,000 live traffic experiments to test out new features and improvements to Search. If you compare that with how many launches actually happened (around 3600, remember?), you can see that only the best and most useful improvements make it into Search.

Always improving

While our search results will never be perfect, these research and evaluation processes have proven to be very effective over the past two decades. They allow us to make frequent improvements and ensure that the changes we make represent the needs of people around the world coming to Search for information.

When you come to Search with a query in mind, you trust that Google will find a number of relevant and helpful pages to choose from. We put a lot of time and effort into improving our search systems to ensure that’s the case.

Working on improvements to our language understanding and other search systems is only part of why Google remains so helpful. Equally important is our ability to fight spam. Without our spam-fighting systems and teams, the quality of Search would be reduced--it would be a lot harder to find helpful information you can trust. 

With low quality pages spamming their way into the top results, the greater the chances that people could get tricked by phony sites trying to steal personal information or infect their computers with malware. If you’ve ever gone into your spam folder in Gmail, that’s akin to what Search results would be like without our spam detection capabilities.

Every year we publish a Webspam Report that details the efforts behind reducing spam in your search results and supporting the community of site creators whose websites we help you discover. To coincide with this year’s report, we wanted to give some additional context for why spam-fighting is so important, and how we go about it.

Defining “spam”

We’ve always designed our systems to prioritize the most relevant and reliable webpages at the top. We publicly describe the factors that go into our ranking systems so that web creators can understand the types of content that our systems will recognize as high quality.

We define “spam” as using techniques that attempt to mimic these signals without actually delivering on the promise of a high quality content, or other tactics that might prove harmful to searchers.

Our Webmaster Guidelines detail the types of spammy behavior that is discouraged and can lead to a lower ranking: everything from scraping pages and keyword stuffing to participating in link schemes and implementing sneaky redirects. 

Fighting spam is never-ending battle, a constant game of cat-and-mouse against existing and new spammy behaviors. This threat of spam is why we’ve continued to be very careful about how much detail we reveal about how our systems work. However, we do share a lot, including resources that provide transparency about the positive behaviors creators should follow to create great information and gain visibility and traffic from Search.

Spotting the spammers

The first step of fighting spam is detection. So how do we spot it? We employ a combination of manual reviews by our analysts and a variety of automated detection systems.

We can’t share the specific techniques we use for spam fighting because that would weaken our protections and ultimately make Search much less useful. But we can share about spammy behavior that can be detected systematically. 

After all, a low quality page might include the right words and phrases that match what you searched for, so our language systems wouldn’t be able to detect unhelpful pages from content alone. The telltale signs of spam are in the behavioral tactics used and how they try to manipulate our ranking systems against our Webmaster Guidelines. 

Our spam-fighting systems detect these behaviors so we can tackle this problem at scale. In fact, the scale is huge. Last year, we observed that more than 25 billion of the pages we find each day are spammy. (If each of those pages were a page in a book, that would be more than 20 million copies of “War & Peace” each day!) This leads to an important question: once we find all this spam, what happens next?

Stopping the spammers

When it comes to how we handle spam, it depends on the type of spam and how severe the violation is. For most of the 25 billion spammy pages detected each day, we’re able to automatically recognize their spammy behavior and ensure they don’t rank well in our results. But that’s not the case for everything. 

As with anything, our automated systems aren’t perfect. That’s why we also supplement them with human review, a team that does its own spam sleuthing to understand if content or sites are violating our guidelines. Often, this human review process leads to better automated systems. We look to understand how that spam got past our systems and then work to improve our detection, so that we catch the particular case and automatically detect many other similar cases overall.

In other cases, we may issue what’s called a manual action, when one of our human spam reviewers finds that content that isn’t complying with our Webmaster Guidelines. This can lead to a demotion or a removal of spam content from our search results, especially if it’s deemed to be particularly harmful, like a hacked site that has pages distributing malware to visitors.

When a manual action takes place, we send a notice to the site owner via Search Console, which webmasters can see in their Manual Actions Report. We send millions of these notices each year, and it gives site owners the opportunity to fix the issue and submit for reconsideration. After all, not all “spam” is purposeful, so if a site owner has inadvertently tried tactics that run afoul of our guidelines, or if their site has been compromised by hackers, we want to ensure they can make things right and have their useful information again available to people in Search. This brings us back to why we invest so much effort in fighting spam: so that Search can bring you good, helpful and safe content from sites across the web.

Discovering great information

It’s unfortunate that there’s so much spam, and so much effort that has to be spent fighting it. But that shouldn’t overshadow the fact there are millions upon millions of businesses, publishers and websites with great content for people to discover. We want them to succeed, and we provide tools, support and guidance to help.

We publish our own Search Engine Optimization Starter Guide to provide tips on how to succeed with appropriate techniques in Search. Our Search Relations team conducts virtual office hours, monitors our Webmaster Community forums, and (when possible!) hosts and participates in events around the world to help site creators improve their presence in Search. We provide a variety of support resources, as well as the Search Console toolset to help creators with search.

We’d also encourage anyone to visit our How Google Search Works site, which shares more generally about how our systems work to generate great search results for everyone.

Sometimes Google Search will show special boxes with information about people, places and things. We call these knowledge panels. They’re designed to help you quickly understand more about a particular subject by surfacing key facts and to make it easier to explore a topic in more depth. Information within knowledge panels comes from our Knowledge Graph, which is like a giant virtual encyclopedia of facts. In this post, we’ll share more about how knowledge panels are automatically generated, how data for the Knowledge Graph is gathered and how we monitor and react to reports of incorrect information.

What’s a knowledge panel?

Our systems aim to show the most relevant and popular information for a topic within a knowledge panel. Because no topic is the same, exactly what is shown in a knowledge panel will vary. But typically, they’ll include:

• Title and short summary of the topic
• A longer description of the subject
• A picture or pictures of the person, place or thing
• Key facts, such as when a notable figure was born or where something is located
• Links to social profiles and official websites

Knowledge panels might also include special information related to particular topics. For example:

• Songs from musical artists
• Upcoming episodes from TV shows
• Rosters of sports teams.

Sources of information for the Knowledge Graph

The information about an “entity”—a person, place or thing—in our knowledge panels comes from our Knowledge Graph, which was launched in 2012. It’s a system that understands facts and information about entities from materials shared across the web, as well as from open source and licensed databases. It has amassed over 500 billion facts about five billion entities.

Wikipedia is a commonly-cited source, but it’s not the only one. We draw from hundreds of sources from across the web, including licensing data that appears in knowledge panels for music, sports and TV. We work with medical providers to create carefully vetted content for knowledge panels for health issues. We also draw from special coding that content owners can use, such as to indicate upcoming events.

On mobile, multiple knowledge panels provide facts

When we first launched knowledge panels, most search activity happened on desktop, where there was room to easily show knowledge panels alongside search results. Today, most search activity happens on mobile, where screen size doesn’t allow for a side-by-side display.

To this end, information from the Knowledge Graph is often not presented through a single knowledge panel on mobile. Instead, one or more knowledge panels may appear interspersed among the overall results.

How we work to improve the Knowledge Graph

Inaccuracies in the Knowledge Graph can occasionally happen. Just as we have automatic systems that gather facts for the Knowledge Graph, we also have automatic systems designed to prevent inaccuracies from appearing. However, as with anything, the systems aren’t perfect. That’s why we also accept reports from anyone about issues.

Selecting the “Feedback” link at the bottom of a knowledge panel or the three dots at the top of one on mobile brings up options to provide feedback to us:

We analyze feedback like this to understand how any actual inaccuracies got past our systems, so that we can make improvements generally across the Knowledge Graph overall. We also remove inaccurate facts that come to our attention for violating our policies, especially prioritizing issues relating to public interest topics such as civic, medical, scientific, and historical issues or where there’s a risk of serious and immediate harm.

How entities can claim and suggest changes to a knowledge panel

Many knowledge panels can be “claimed” by the subject they are about, such as a person or a company. The claiming process—what we call getting verified—allows subjects to provide feedback directly to us about potential changes or to suggest things like a preferred photo. For local businesses, there’s a separate process of claiming that operates through Google My Business. This enables local businesses to manage special elements in their knowledge panels, such as opening hours and contact phone numbers.

For more information about topics like this, check out our How Search Works blog series and website.