Monthly Archives: August 2011

Google News now crawling with Googlebot

Posted by David Smydra, Google News Product Specialist

(Cross-posted on the Webmaster Central Blog)

Google News recently updated our infrastructure to crawl with Google’s primary user-agent, Googlebot. What does this mean? Very little to most publishers. Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.

Our Help Center provides detailed guidance on using the robots exclusion protocol for Google News, and publishers can contact the Google News Support Team if they have any questions, but we wanted to first clarify the following:
  • Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies. (You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.)
  • Your analytics tool will still be able to differentiate user traffic coming to your website from Google Search and traffic coming from Google News, so you should see no changes there. The main difference is that you will no longer see occasional automated visits to your site from the Googlebot-news crawler.
  • If you’re currently respecting webmaster guidelines for Googlebot, you will not need to make any code changes to your site. Sites that have implemented subscriptions using a metered model or who have implemented First Click Free will not experience any changes. For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page. Our Webmaster Guidelines provide additional information about “cloaking” (i.e., showing a bot a different version than what users experience). Learn more about Google News and subscription publishers in this Help Center article.
  • Rest assured, your Sitemap will still be crawled. This change does not affect how we crawl News Sitemaps. If you are a News publisher who hasn’t yet set up a News Sitemap and are interested in getting started, please follow this link.
  • For any publishers that wish to opt out of Google News and stay in Google Search, you can simply disallow Googlebot-news and allow Googlebot. For more information on how to do this, consult our Help Center.
As with any website, from time to time we need to make updates to our infrastructure. At the same time, we want to continue to provide as much control as possible to news web sites. We hope we have answered any questions you might have about this update. If you have additional questions, please check out our Help Center.

Google-hosted Custom Search Engines now use the Element

Two months ago we announced that all Google-hosted Custom Search Engines would be upgraded to use the Element. We’re happy to report that that transition is now complete for all Custom Search users. Those that visit the Control Panel will find better tools for customizing the presentation of their CSEs.

Moving to the Element enables us to quickly deliver new features that Custom Search users can easily turn on for their sites and we will soon upgrade all iframe results to use embedded Elements.

As always, we look forward to your feedback.

Posted by: Liang Ch'ng, Software Engineer

Enhancements to Google News for Android tablets and iPads

Posted by Arun Prasath, Tech Lead, Mobile Google News

Alongside working on improving the Google News design for smartphones, we have also been looking into enhancing our offering for tablet devices. Today, we are launching a few minor enhancements to Google News for Android tablets and iPads.

We have optimized columns in the home page and section pages so that they can be more easily viewed in portrait and landscape orientations. Some of the other updates include: a conveniently placed menu on the top for navigating across sections, support of finger swiping through the multimedia strip in expanded story boxes and a more friendly edition picker.


These improvements are now available in the U.S. edition. To view these changes, just type news.google.com on the browser of your favorite tablet.

Google News highlights unique content with Editors’ Picks

(Cross-posted on the Official Google Blog)

News organizations tell stories online in ways that bring together the best of traditional and digital journalism, whether that involves long-form investigative features, compelling photo slideshows or interactive maps and charts that add new levels of engagement to the day's news. To help connect you to the best works of news publishers, Google News is introducing a new section in the right-hand column of the U.S. edition. The section is called "Editors' Picks,” and it displays original content that publishers have selected as highlights from their publications. This is the latest addition to recent improvements we’ve made to the variety and presence of stories and multimedia on Google News.

An array of news organizations, including local, national and niche publishers, are now using Editors’ Picks to display their best, most engaging content. Because Google News relies on algorithms, Editors' Picks will always be just that—picks provided by publishers themselves, and not by Google. You can browse a set of publisher feeds that span national, specific and local interests—like The Atlantic, The Wall Street Journal, ProPublica, the Guardian and The Root, among many others—via the side-to-side arrows next to each publisher's logo. The feeds you see are chosen based on a variety of factors, including your news preferences. If you’re interested in using source preferences on Google News, Editors' Picks helps you do that with the slider that appears just below the articles.


You may have first noticed Editors’ Picks as an experiment last year. Based on the data from that experiment, we have been working with nearly two dozen publishers in recent months and have seen a positive response from readers and publishers alike: readers get the news they're interested in from the sources they trust, and publishers receive higher traffic to their websites. We encourage any news organizations that are interested to visit our Help Center to get started.

Posted by Yogita Mehta, Software Engineer, Google News Team

Introducing Google Tasks Porter

We're happy to announce a new open source application that allows you to import and export your Google Tasks, called Google Tasks Porter.

Google Tasks Porter is designed with other applications that contain task lists in mind. It supports import and export via the iCalendar format which is used by a variety of applications, including iCalendar itself. It also supports import and export to Microsoft Outlook via a CSV format. Additionally, Google Tasks Porter supports export from Remember the Milk using its iCalendar export, and import to Remember the Milk via email. You can also download a list of all your tasks in an HTML format which is designed to be portable and parseable.

Google Tasks Porter allows you to create a set of "snapshots" of your data, each representing a list of all your tasks at a particular point in time. You can then save or delete these snapshots, and you can export a snapshot at any time to another application using any of the available formats. You can also upload an ics or csv file in order to import the tasks contained therein into Google Tasks.

The application is available on the web at http://google-tasks-porter.appspot.com. The source code is also available on Google Code at http://google-tasks-porter.googlecode.com using the Apache License, Version 2.0. Please try the application out and let us know what you think. You can provide feedback via the mailing list at [email protected].

By Dwight Guth, The Data Liberation Front

Source: Data Liberation