At Google, we like to say that Search is not a solved problem: We’re constantly making improvements (more than 4,800 last year alone). These changes can be big launches or small tune-ups, but they’re all designed to make Search work better for you, and to make sure you can find relevant, high quality information when you need it.

One of the key ways we determine if an improvement to Search works well is through the help of search quality raters. This group of more than 10,000 people all over the world work from a common set of search quality rater guidelines used to evaluate the quality of search results — which are publicly available. Today, we wanted to give you an idea of how these guidelines work, and how — just like Search itself — they improve over time.

What are the search quality rater guidelines?

The quality rater guidelines are more than 170 pages long, but if we have to boil it down to one phrase, we’d say they help make sure Search is returning relevant results from the most reliable sources available.

Information quality is at the heart of Search, and our systems fundamentally work to surface high-quality information. The rater guidelines help raters determine if a planned improvement is meeting that goal by providing a clear, uniform definition that all raters use to assess the results they see.

More specifically, high-quality information is content which demonstrates expertise, authoritativeness and trustworthiness on a topic, or E-A-T for short. For example, a health site with content from doctors and produced by a medical institution would have a high level of what many would consider to be expertise, authoritativeness and trustworthiness. The rater guidelines also define low-quality content on the web, such as content that spreads hate or seeks to deceive users.

Who uses the guidelines?

As we noted, the changes we make to Search are rigorously tested and evaluated by real people. Our search quality raters provide us with insights and evaluate pages against our guidelines to help make sure our systems — and proposed improvements — are working as intended.

What that looks like in practice is often a “side-by-side” test where a rater will look at two sets of Search results, one from the current version of Google and the other from an improvement we’re testing. Raters will review the pages in each set of results, and evaluate if the pages are a helpful match for the query based on our rater guidelines.

The ratings they provide don’t directly impact how a page or site appears in Search. Instead, they help us measure how well our systems are working to deliver great content.

How often are the rater guidelines updated?

Just like we make improvements to Search, we update the rater quality guidelines from time to time to make sure they’re working as intended.

Some changes are meant to tackle issues we’ve identified in Search, and may include expanded sections and new examples to help guide raters. For example, in 2017, we updated our guidelines to provide more detailed examples of low-quality web pages that included misleading information, unexpected offensive results, hoaxes or other content.

Sometimes, we identify concepts that are especially challenging for raters. We then make changes to the guidelines to improve ratings. In 2020, for instance, we provided new guidance on how to tell if a result from a dictionary or encyclopedia would be useful for a certain query.

Other changes focus on things like refreshing the language for clarity and updating organization. That’s what made up most of our October 2021 update, which included clarifications of what constitutes lowest quality content, and refreshed and modernized guidance on researching the reputation of websites.

We rigorously review, test and evaluate all changes to ensure they’re helpful and having the intended effect. And we have a publicly available log at the end of our guidelines describing in detail any changes we make. Ultimately, these updates are designed to make Search work better for you.

When you come to Google and do a search, there might be billions of pages that are potential matches for your query, and millions of new pages being produced every minute. In the early days, we updated our search index once per month. Now, like other search engines Google, is constantly indexing new info to make accessible through Search.

But to make all of this information useful, it’s critical that we organize it in a way that helps people quickly find what they’re looking for. With this in mind, here’s a closer look at how we approach organizing information on Google Search.

Organizing information in rich and helpful features

Google indexes all types of information--from text and images in web pages, to real-world information, like whether a local store has a sweater you’re looking for in stock. To make this information useful to you, we organize it on the search results page in a way that makes it easy to scan and digest. When looking for jobs, you often want to see a list of specific roles. Whereas if you’re looking for a restaurant, seeing a map can help you easily find a spot nearby. 

We offer a wide range of features--from video and news carousels, to results with rich imagery, to helpful labels like star reviews--to help you navigate the available information more seamlessly. These features include links to web pages, so you can easily click to a website to find more information. In fact, we’ve grown the average number of outbound links to websites on a search results page from only 10 (“10 blue links”) to now an average of 26 links on a mobile results page. As we’ve added more rich features to Google Search, people are more likely to find what they’re looking for, and websites have more opportunity to appear on the first page of search results.

Presenting information in rich features, like an image carousel or a map, makes Google Search more helpful, both to people and to businesses. These features are designed so you can find the most relevant and useful information for your query. By improving our ability to deliver relevant results, we’ve seen that people are spending more time on the webpages they find through Search. The amount of time spent on websites following a click from Google Search has significantly grown year over year. 

Helping you explore and navigate topics

Another important element of organizing information is helping you learn more about a topic. After all, most queries don’t just have a single answer--they’re often open-ended questions like “dessert ideas.”

Our user experience teams spend a lot of time focused on how we can make it easy and intuitive to refine your search as you go. This is why we’ve introduced features like carousels, where you can easily swipe your phone screen to get more results. For instance, if you search for “meringue”, you might see a list of related topics along with related questions that other people have asked to help you on your journey.

How features and results are ranked

Organizing information into easy-to-use formats is just one piece of the puzzle. To make all of this information truly useful, we also must order, or “rank,” results in a way that ensures the most helpful and reliable information rises to the top.

Our ranking systems consider a number of factors--from what words appear on the page, to how fresh the content is--to determine what results are most relevant and helpful for a given query. Underpinning these systems is a deep understanding of information--from language and visual content to context like time and place--that allows us to match the intent of your query with the most relevant, highest quality results available. 

An important thing to remember is that ranking is dynamic. New things are always happening in the world, so the available information and the meaning of queries can change day-by-day. This summer, searches for “why is the sky orange” turned from a general question about sunsets to a specific, locally relevant query about weather conditions on the West Coast of the U.S. due to wildfires. We constantly evaluate the quality of our results to ensure that even as queries or content changes, we’re still providing helpful information. More than 10,000 search quality raters around the world help us conduct hundreds of thousands of tests every year, and it’s through this process that we know that our investments in Google Search truly benefit people.

We’ve heard people ask if we design our search ranking systems to benefit advertisers, and we want to be clear: that is absolutely not the case. We never provide special treatment to advertisers in how our search algorithms rank their websites, and nobody can pay us to do so. 

Ongoing investment in a high quality experience

As we’ve seen for many years, and as was particularly apparent in the wake of COVID, information needs can change rapidly. As the world changes, we are always looking for new ways we can make Google Search better and help people improve their lives through access to information.

Every year, we make thousands of improvements to Google Search, all of which we test to ensure they’re truly making the experience more intuitive, modern, delightful, helpful and all-around better for the billions of queries we get every day. Search will never be a solved problem, but we’re committed to continuing to innovate to make Google better for you.

Since Google was founded more than 22 years ago, we’ve continued to pursue an ambitious mission of organizing the world’s information and making it universally accessible and useful. While we started with organizing web pages, our mission has always been much more expansive. We didn’t set out to organize the web’s information, but all the world’s information. 

Quickly, Google expanded beyond the web and began to look for new ways to understand the world and make information and knowledge accessible for more people. The internet--and the world--have changed a lot since those early days, and we’ve continued to improve Google Search to both anticipate and respond to the ever-evolving information needs that people have. 

It’s no mystery that the search results you saw back in 1998 look different than what you might find today. So we wanted to share an overview of where the information on Google comes from and, in another post, how we approach organizing an ever-expanding universe of web pages, images, videos, real-world insights and all the other forms of information out there.

Information from the open web

You’re probably familiar with web listings on Google--the iconic “blue link” results that take you to pages from across the web. These listings, along with many other features on the search results page, link out to pages on the open web that we’ve crawled and indexed, following instructions provided by the site creators themselves.

Site owners have the control to tell our web crawler (Googlebot) what pages we should crawl and index, and they even have more granular controls to indicate which portions of a page should appear as a text snippet on Google Search. Using our developer tools, site creators can choose if they want to be discovered via Google and optimize their sites to improve how they’re presented, with the aim to get more free traffic from people looking for the information and services they’re offering. 

Google Search is one of many ways people find information and websites.  Every day, we send billions of visitors to sites across the web, and the traffic we send has grown every year since Google started. This traffic goes to a wide range of websites, helping people discover new companies, blogs, and products, not just the largest, well known sites on the web. Every day, we send visitors to well over 100 million different websites. 

Common knowledge and public data sources

Creators, publishers and businesses of all sizes work to create unique content, products and services. But there is also information that falls into the category of what you might describe as common knowledge--information that wasn’t uniquely created or doesn’t “belong” to any one person, but represents a set of facts that is broadly known. Think: the birthdate of a historical figure, the height of the tallest mountain in South America, or even what day it is today. 

We help people easily find these types of facts through a variety of Google Search features like knowledge panels. The information comes from a wide range of openly licensed sources such as Wikipedia, The Encyclopedia of Life, Johns Hopkins University CSSE COVID-19 Data, and the Data Commons Project, an open knowledge database of statistical data we started in collaboration with the U.S. Census, Bureau of Labor Statistics, Eurostat, World Bank and many others.

Another type of common knowledge is the product of calculations, and this is information that Google often generates directly. So when you search for a conversion of time (“What time is it in London?”) or measurement (“How many pounds in a metric ton?”), or want to know the square root of 348, those are pieces of information that Google calculates. Fun fact: we also calculate the sunrise and sunset times for locations based on latitude and longitude!

Licenses and partnerships

When it comes to organizing information, unstructured data (words and phrases on web pages) is more challenging for our automated systems to understand. Structured databases, including public knowledge bases like Wikidata, make it a lot easier for our systems to understand, organize and present facts in helpful features and formats.

For some specialized types of data, like sports scores, information about TV shows and movies, and song lyrics, there are providers who work to organize information in a structured format and offer technical solutions (like APIs) to deliver fresh info. We license data from these companies to ensure that providers and creators (like music publishers and artists) are compensated for their work. When people come to Google looking for this information, they can access it right away.

We always work to deliver high quality information, and for topics like health or civic participation that affect people’s livelihoods, easy access to reliable, authoritative information is critically important. For these types of topics, we work with organizations like local health authorities, such as the CDC in the U.S., and nonpartisan, nonprofit organizations like Democracy Works to make authoritative information readily available on Google.

Information that people and businesses provide

There’s a wide range of information that exists in the world that isn’t currently available on the open web, so we look for ways to help people and businesses share these updates, including by providing information directly to Google. Local businesses can claim their Business Profile and share the latest with potential customers on Search, even if they don’t have a website. In fact, each month Google Search connects people with more than 120 million businesses that don’t have a website. On average, local results in Search drive more than 4 billion connections for businesses every month, including more than 2 billion visits to websites as well as connections like phone calls, directions, ordering food and making reservations.

We’re also deeply investing in new techniques to ensure that we’re reflecting the latest accurate information. This can be especially challenging as local information is constantly changing and not often accurately reflected on the web. For example, in the wake of COVID-19, we’ve used our Duplex conversational technology to call businesses, helping to update their listings by confirming details like modified store hours or whether they offer takeout and delivery. Since this work began, we’ve made over 3 million updates to businesses like pharmacies, restaurants and grocery stores that have been seen over 20 billion times in Maps and Search. 

Other businesses like airlines, retailers and manufacturers also provide Google and other sites with data about their products and inventory through direct feeds. So when you search for a flight from Bogota to Lima, or want to learn more about the specs of the hottest new headphones, Google can provide high quality information straight from the source.

We also provide ways for people to share their knowledge about places across more than 220 countries and territories. Thanks to millions of contributions submitted by users every day--from reviews and ratings to photos, answers to questions, address updates and more--people all around the world can find the latest, accurate local information on Google Search and Maps. 

Newly created information and insights from Google

Through advancements in AI and machine learning, we’ve developed innovative ways to derive new insights from the world around us, providing people with information that can not only help them in their everyday lives, but also keep them safe.

For years, people have turned to our Popular Times feature to help gauge the crowds at their favorite brunch spots or visit their local grocery store when it’s less busy. We're continually improving the accuracy and coverage of this feature, currently available for 20 million places around the world on Maps and Search. Now, this technology is serving more critical needs during COVID. With an expansion of our live busyness feature, these Google insights are helping people take crowdedness into account as they patronize businesses through the pandemic. 

We also generate new insights to aid in crisis response--from wildfire maps based on satellite data to AI-generated flood forecasting--to help people stay out of harm’s way when disaster strikes.

Organizing information and making it accessible and useful

Simply compiling a wide range of information is not enough. Core to making information accessible is organizing it in a way that people can actually use it. 

You come to Google with an idea of what you’d like to search for. As soon as you start typing, predictions appear in the search box to help you finish what you’re typing. These time-saving predictions are from a feature called Autocomplete, which we covered previously in this How Search Works series.

In this post, we’ll explore how Autocomplete’s predictions are automatically generated based on real searches and how this feature helps you finish typing the query you already had in mind. We’ll also look at why not all predictions are helpful, and what we do in those cases.

Where predictions come from

Autocomplete predictions reflect searches that have been done on Google. To determine what predictions to show, our systems begin by looking at common and trending queries that match what someone starts to enter into the search box. For instance, if you were to type in “best star trek…”, we’d look for the common completions that would follow, such as “best star trek series” or “best star trek episodes.”

That’s how predictions work at the most basic level. However, there’s much more involved. We don’t just show the most common predictions overall. We also consider things like the language of the searcher or where they are searching from, because these make predictions far more relevant. 

Below, you can see predictions for those searching for “driving test” in the U.S. state of California versus the Canadian province of Ontario. Predictions differ in naming relevant locations or even spelling “centre” correctly for Canadians rather than using the American spelling of “center.”

To provide better predictions for long queries, our systems may automatically shift from predicting an entire search to portions of a search. For example, we might not see a lot of queries for “the name of the thing at the front” of some particular object. But we do see a lot of queries for “the front of a ship” or “the front of a boat” or “the front of a car.” That’s why we’re able to offer these predictions toward the end of what someone is typing.

We also take freshness into account when displaying predictions. If our automated systems detect there’s rising interest in a topic, they might show a trending prediction even if it isn’t typically the most common of all related predictions that we know about. For example, searches for a basketball team are probably more common than individual games. However, if that team just won a big face-off against a rival, timely game-related predictions may be more useful for those seeking information that’s relevant in that moment.

Predictions also will vary, of course, depending on the specific topic that someone is searching for. People, places and things all have different attributes that people are interested in. For example, someone searching for “trip to New York” might see a prediction of “trip to New York for Christmas,” as that’s a popular time to visit that city. In contrast, “trip to San Francisco” may show a prediction of “trip to San Francisco and Yosemite.” Even if two topics seem to be similar or fall into similar categories, you won’t always see the same predictions if you try to compare them.  Predictions will reflect the queries that are unique and relevant to a particular topic.

Overall, Autocomplete is a complex time-saving feature that’s not simply displaying the most common queries on a given topic. That’s also why it differs from and shouldn’t be compared against Google Trends, which is a tool for journalists and anyone else who’s interested to research the popularity of searches and search topics over time.

Predictions you likely won’t see

Predictions, as explained, are meant to be helpful ways for you to more quickly finish completing something you were about to type. But like anything, predictions aren’t perfect. There’s the potential to show unexpected or shocking predictions. It’s also possible that people might take predictions as assertions of facts or opinions. We also recognize that some queries are less likely to lead to reliable content.

We deal with these potential issues in two ways. First and foremost, we have systems designed to prevent potentially unhelpful and policy-violating predictions from appearing. Secondly, if  our automated systems don’t catch predictions that violate our policies, we have enforcement teams that remove predictions in accordance with those policies.

Our systems are designed to recognize terms and phrases that might be violent, sexually-explicit, hateful, disparaging or dangerous. When we recognize that such content might surface in a particular prediction, our systems prevent it from displaying. 

People can still search for such topics using those words, of course. Nothing prevents that. We’re simply not wanting to unintentionally shock or surprise people with predictions they might not have expected.

Using our automated systems, we can also recognize if a prediction is unlikely to return much reliable content. For example, after a major news event, there can be any number of unconfirmed rumors or information spreading, which we would not want people to think Autocomplete is somehow confirming. In these cases, our systems identify if there’s likely to be reliable content on a particular topic for a particular search. If that likelihood is low, the systems might automatically prevent a prediction from appearing. But again, this doesn’t stop anyone from completing a search on their own, if they wish.

While our automated systems typically work very well, they don’t catch everything. This is why we have policies for Autocomplete, which we publish for anyone to read. Our systems aim to prevent policy-violating predictions from appearing. But if any such predictions do get past our systems, and we’re made aware (such as through public reporting options), our enforcement teams work to review and remove them, as appropriate. In these cases, we remove both the specific prediction in question and often use pattern-matching and other methods to catch closely-related variations.

As an example of all this in action, consider our policy about names in Autocomplete, which began in 2016. It’s designed to prevent showing offensive, hurtful or inappropriate queries in relation to named individuals, so that people aren’t potentially forming an impression about others solely off predictions.  We have systems that aim to prevent these types of predictions from showing for name queries. But if violations do get through, we remove them in line with our policies. 

You can always search for what you want

Having discussed why some predictions might not appear, it’s also helpful to remember that predictions are not search results. Occasionally, people concerned about predictions for a particular query might suggest that we’re preventing actual search results from appearing. This is not the case. Autocomplete policies only apply to predictions. They do not apply to search results. 

We understand that our protective systems may prevent some useful predictions from showing. In fact, our systems take a particularly cautious approach when it comes to names and might prevent some non-policy violating predictions from appearing. However, we feel that taking this cautious approach is best. That’s especially because even if a prediction doesn’t appear, this does not impact the ability for someone to finish typing a query on their own and finding search results. 

We hope this has helped you understand more about how we generate predictions that allow you to more quickly complete the query you started, whether that’s while typing on your laptop or swiping the on-screen keyboard on your phone.

On the morning of September 10, millions of people in Northern California woke up to an orange sky after wildfire smoke spread like a thick layer across the West Coast. It persisted for days, and it was the first time lots of people had ever seen something like this. 

To understand what was happening, many people turned to Search. According to Google Trends, searches for “why is the sky orange” hit an all-time high this month in the United States. As you can see in the graph below, this wasn't a totally new query. There are many pages on the web with general scientific explanations of what can cause the sky to turn orange. But people wanted to know why, in that moment, where they were, the sky was tangerine tinted.

So how does Google respond to a query spike like this? Well,language understanding is at the core of Search, but it’s not just about the words. Critical context, like time and place, also helps us understand what you’re really looking for. This is particularly true for featured snippets, a feature in Search that highlights pages that our systems determine are likely a great match for your search. We’ve made improvements to better understand when fresh or local information -- or both -- is key to delivering relevant results to your search. 

Put simply, instead of surfacing general information on what causes a sunset, when people searched for “why is the sky orange” during this time period, our systems automatically pulled in current, location-based information to help people find the timely results they were searching for. 

Over the course of the week, we saw even more examples of these systems at work. As a residual effect of the wildfires, New York City and Massachusetts started experiencing a hazy sky. But that wasn’t the case in all states. So for a query like “why is it hazy?” local context was similarly important for providing a relevant result.

For this query, people in New York found an explanation of how the wildfire smoke was caught in a jet stream, which caused the haze to move east. People in Boston would have found a similar feature snippet, but specific to the conditions in that city. And those in Alaska, who were not impacted, would not see these same results. 

These are just two of billions of queries we get each day, and as new searches arise and information in the world changes, we’ll continue to provide fresh, relevant results in these moments.