Monthly Archives: May 2018

Troubleshooting tips: How to talk so your cloud provider will listen (and understand)

Editor’s note: We’re excited to bring you this blog post from the team of Google experts who wrote the book (really!) on Site Reliability Engineering (SRE) a few years back. The second edition of the book is underway, and as a teaser, this post delves into one area of SRE that’s relevant to many IT teams today: troubleshooting in the age of cloud computing. This is part one of two. Then, check out the second installment specifically on troubleshooting cloud provider communications.

Effective technology troubleshooting requires a systematic approach, as opposed to luck or experience. Troubleshooting can be a learned skill, as discussed in our site reliability engineering (SRE) troubleshooting primer.

But how does that change when you and your operations team are running systems and services on cloud infrastructure? Regardless of where your websites or apps live, you’re the one getting paged when the site goes down and you are the one under pressure to solve the problems and answer questions.

Cloud presents a new way of working for IT teams shifting away from legacy systems. You had full visibility and control of all aspects of your system when it was on-premises, but now you depend on off-site, cloud-based infrastructure, into which you may have limited visibility or understanding. That’s true no matter how top-notch your systems and processes are and how great a troubleshooter you are. You simply can’t see much into many cloud provider systems. The troubleshooting challenge is no longer limited to pure debugging; you also need to effectively communicate with your cloud provider. You’ll need to engage each provider’s support process and engineers to find where problems originated and fix them as soon as possible. This is an area of opportunity for IT teams as they gain new skills and adapt to the cloud-based technology model.

It’s definitely possible to do cloud troubleshooting right. When you’re choosing a provider, make sure to understand how they’ll work with you during any issues. Once you’re working with a provider, you have more control than you think over how you communicate issues and speed along their resolution. We propose this model, inspired by the SRE model of actionable improvements, for working with your cloud provider to troubleshoot more effectively and efficiently. (For more on cloud provider communications throughout the troubleshooting process, see the companion post.)

Understand your cloud provider's support workflow

Your goal here is to figure out the best way to provide the right information to your provider when an issue inevitably arises. This is a useful place to start, especially since you may depend on multiple cloud providers to run your infrastructure. Your interaction with cloud support will typically begin with questions related to migration. Your relationship then progresses into the domain of production integration, and finally, joint troubleshooting of production problems.

Keep in mind that different cloud providers have different philosophies when it comes to customer interaction. Some provide a large degree of freedom and little direct support. They expect you to find answers to most of your questions from online forums such as Stack Overflow. Other providers emphasize tight customer integration and joint troubleshooting. You have some homework to do before you begin serving real customer traffic from your cloud deployment. Talk to your cloud provider's salespeople, project managers and support engineers, to get a sense of how they approach support. Ask them the following questions:
  • What does the lifecycle of a typical support issue report look like?
  • What is your internal escalation process if an issue becomes complex or critical?
  • Do you have an internal SLO for <service name>? If so, what is that SLO?
  • What types of premium support are available?
This step is critical in reducing your frustration when you have to troubleshoot an issue that involves a cloud provider’s giant black box.

Communicate with cloud provider support efficiently

Once you have a sense of the provider’s support workflow, you can figure out the best way to get your information across. There are some best practices for filing the perfect issue report with cloud provider support teams, including what to say in your issue report and why. These guidelines follow the 80/20 rule: we try to give 20% of the details that will be useful in 80% of your issue reports. The same principles apply if you're filing bug reports in issue trackers or posting to user groups and forums.

Your guiding principle when communicating to cloud providers should be clarity: specify the appropriate level of technical detail and communicate expectations explicitly.

Provide basic information
It may seem like common sense, but these basics are essential to include in an issue report. Failing to provide any of these details leads to delays and a poor experience.

Include four critical details
Effective troubleshooting starts with time, product, location and specific identifiers.

1. Time
Here are a few examples of including time in an issue report:
  • Starting at 2017-09-08 15:13 PDT and ending 5 minutes later, we observed...
  • Observed intermittently, starting no earlier than 2017-09-10 and observed 2-5 times...
  • Ongoing since 2017-09-08 15:13 PDT...
  • From 2017-09-08 15:13 PDT to 2017-09-08 22:22 PDT...
Including the onset time and duration allow support teams to focus their time-series monitoring on the relevant period. Be explicit about whether the issue is ongoing or whether it was observed only in the past. If the issue is not ongoing, be explicit about that fact and provide an end time, if possible.

Remember to always include the time zone. ISO 8601 format is a good choice because it is unambiguous and easy to sort. If you instead specify time in a relative way, whomever you're working with must convert local time into an absolute format they can input into time-series monitoring tools. This conversion is error-prone and costly: for example, sending an email about something that happened "earlier yesterday" means that your counterpart has to search for an email header and start doing mental math. This introduces cognitive load, which decreases the person’s mental energy available for solving the technical problem.

If an issue was intermittent over some period of time, state when it was first observed, or perhaps note a time in the past when it was surely not happening. Include the frequency of observations and note one or two specific examples.

Meanwhile, here are some antipatterns to avoid:
  • Earlier today: Not specific enough
  • Yesterday: Requires the recipient to figure out the implied date; can be confusing especially when work crosses the International Date Line
  • 9/8: Ambiguous, as the date might be interpreted as September in United States, or August in other locales. Use ISO 8601 format for clarity.
2. Product
Be as specific as possible in your issue report about the product you're using, including version information where applicable. These, for example, aren’t specific enough to locate the components or logs that can help with diagnosis:
  • REST API returned errors...
  • The data mining query interface is hanging...
Ideally, you should refer to specific APIs or URLs, or include screenshots. If the issue originates in a specific component or tool (for example, the CLI or Terraform), clearly identify that tool. If multiple products are involved, be specific about each one.Describe the behavior you're observing, and the behavior you expected to occur.

  • Can't create virtual machine: It's not clear how you're attempting to create those machine, nor does it say what the failure mode is.
  • The CLI command is giving an error:
    • Instead, provide the specific error, and the command syntax so others can run the command themselves.
    • Better: I ran 'mktool create my-instance --zone us-central1' and got the following error message...
3. Location
It's important to specify the region and zone because cloud providers often roll out changes to one region or zone at a time. Therefore, region or zone is a proxy for a cloud-internal software version number.
These are examples of how you might include location information in an issue report:
  • In us-east1-a... 
  • I tried regions eu-west-1 and eu-west-3...
Given this information, the support team can see if there's a rollout underway in a given location, or map your issue to an internal release ID for use in an internal bug report.

4. Specific identifiers
Project identifiers are included in many troubleshooting tools. Specify whether you observed the error in multiple projects, or in one project but not another.

These are examples of specific identifiers:
  • In project 123412341234 or my-project-id... 
  • Across multiple projects (including 123412341234)... 
  • Connecting to cloud external IP from our corporate gateway 
IP addresses are another form of unambiguous identifiers, though they also require additional details when used in an issue report. When specifying an IP, try to describe the context of how it's used. For example, specify whether the IP is connected to a VM instance, a load balancer or a custom route, or if it's an API endpoint. If the IP address isn't part of the cloud platform (for example, your home internet, a VPN endpoint or an external monitoring system), specify that information. Remember, occurs many times in the world.

  • One of our instances is unreachable…: Overly vague 
  • We can't connect from the Internet...: Overly vague 
Note that other models, such as the Five Ws (popular in fields like journalism) can provide structure to your report.

Specify impact and response expectations
Basic information to include in your issue report to a cloud provider should also include how it’s affecting your business and when it needs to be resolved.

Priority expectations
Cloud provider support commonly uses the priority you specify to initially route the issue report and to determine the urgency of the issue. The priority rating drives the speed of response, potentially paging oncall personnel.

In addition to selecting the appropriate priority, it's useful to add a sentence describing the impact. Help avoid incorrect assumptions by being explicit about why you selected P1.

Think of the priority in terms of the impact to your business. Your cloud provider may have what appear to be strict definitions of priority (e.g., P1 signified a total outage). Don't let these definitions slow progress on issues that are business critical; extrapolate impact if your issue isn't addressed, or describe the worst case scenario of an exposure related to the issue. For example, the following two descriptions essentially describe impeding P1 issues:
  • Our backlog is increasing. Current impact is minor, but if this issue is not fixed in 12 hrs, we're effectively down. 
  • A key monitoring component has failed. While there is no current impact, this means we have a blind spot that will cause the next failure to become a user visible outage. 
Response time expectations
If you have specific needs related to response time, indicate them clearly. For example, you might specify "I need a response by 5pm because that's when my shift ends." If you have internal outage communication SLOs, make sure you request a response time from your provider that is within that interval so you can meet those SLOs. Cloud providers likely have 24/7 support, but if this isn't the case, or if your relevant personnel are in a particular time zone, communicate your time zone-specific needs to your provider.

Including these details in your issue report will ideally save you time later and speed up your overall provider resolution process. Check out part two for tips on communicating with your cloud provider throughout the actual troubleshooting process.

Related content:
SRE vs. DevOps
Incident management at Google —adventures in SRE-land
Applying the Escalation Policy — CRE life lessons
Special thanks to Ralph Pearson, J.C. van Winkel, John Lowry, Dermot Duffy and Dave Rensin

Troubleshooting tips: Help your cloud provider help you

Editor’s note: We’re excited to bring you this blog post from the team of Google experts who wrote the book (really!) on Site Reliability Engineering (SRE) a few years back. The second edition of the book is underway, and this post delves into one area of SRE that’s relevant to many IT teams today: troubleshooting in the age of cloud computing. This is part two of two. Check out part one on writing better issue reports for cloud provider support.

Troubleshooting computer systems is an act as old as computers themselves. Some might even call it an art. The cloud computing paradigm entails a fundamental change to how IT teams conduct troubleshooting.

Successful IT troubleshooting doesn’t depend only on luck or experience, but is a deliberate process that can be taught. When you’re using cloud-based infrastructure, you’re often troubleshooting via a cloud provider’s help desk, adding another layer to helping users. Because of this shift away from the traditional IT team model, your communications with the provider are essential. (See part one for more on putting together an effective issue report to improve troubleshooting from the start.)

Once you’ve communicated the issue to your provider, you’ll be working with the provider’s support team to get the issue fixed.

The essentials of cloud troubleshooting

Those diagnosing a technical problem with cloud infrastructure are seeking possible explanations (hypotheses) and evidence that explains the problem. In the short term, they look for changes in the system that roughly correlate with the problem, and consider rolling back, as a first step to mitigate the problem and stop the bleeding. The longer-term goal is to identify and fix the root cause so the problem will not recur.

From the site reliability engineering (SRE) perspective,  the general approach for troubleshooting is as follows:

  • Triage: Mitigate the impact if possible
  • Examine: Gather observations and share them
  • Diagnose: Create a hypothesis that explains the observations
  • Test and treat:
    • Identify tests that may prove or disprove the hypothesis
    • Execute the tests and agree on the meaning of the result
    • Move on to the next hypothesis; repeat until solved

When you’re working with a cloud provider on troubleshooting an issue, there are parts of the process you’re unable to control. But you can follow the steps on your end. Here’s what you can do when submitting a report to your cloud provider support team.

1. Communicate any troubleshooting you've already done
By the time you open an issue report, you've probably done some troubleshooting already. You may have checked the provider’s status page, for example. Share the steps you've taken and any key findings. Keep a timeline and log book of what you have done and share it with the provider. This means that you should start keeping a log book as soon as possible, from the start of detection of your problem. Keep in mind that while cloud providers may have telemetry that provides real-time omniscient awareness of the state of their infrastructure, the dependencies that result from your particular implementation may be less obvious. By design, your particular use of cloud resources is proprietary and private, so your troubleshooting vantage point is vital.

If you think you have a diagnosis, explain how you came to that conclusion. If you think others can reproduce the issue, include the steps to do so. A reproducible test in an issue report usually leads to the fastest resolution.

You may have an idea or guess about what's causing the problem. Be careful to avoid confirmation bias—looking for evidence to support your guess without considering evidence to the contrary.

2. Be specific and explicit about the issue
If you've ever played the telephone game, in which players whisper a message from person to person, you've seen how human translation and interpretation can lead to communication gaps. Rather than describing information in your provider communications, try to share it. Doing so reduces the chance that your reader will misinterpret what you're saying and can help speed up troubleshooting. Don’t assume that your provider has access to all of this information; customer privacy means that they may not, by design.

For example:

  • Use a screenshot to show exactly what you see
  • For web-based interfaces, provide a .HAR (Http ARchive) file
  • Attach information like tcpdump output, logs snippets and example stack traces

3. Report production outages quickly
An issue is considered to be a production outage if your application has stopped serving traffic to users or is experiencing similar business-critical impact. Report production outages to your cloud provider support as soon as possible. Issues that block a small number of developers in a developer test environment are normally not considered production outages, so they should be reported at lower priorities.

Normally, when cloud provider support is alerted about a production outage, they quickly triage the situation with the following steps:

  1. Immediately check for known issues affecting the infrastructure.
  2. Confirm the nature of the issue.
  3. Establish communication channels.

Typically, you can expect a quick response with a brief message, which might contain:

  • Whether or not there is a known issue affecting multiple customers
  • An acknowledgement that they can observe the issue you've reported or a request for more details
  • How they intend to communicate (for example, phone, Skype, or issue report)

It’s important to quickly create an issue report including the four critical details (described in part one  and then begin deeper troubleshooting on your side of the equation. If your organization has a defined incident management process (see Managing Incidents), escalating to your cloud provider should be among your initial steps.

4. Report networking issues with specificity
Most cloud providers’ networks are huge and complex, composed of many technologies and teams. It's important to quickly identify a networking-specific problem as such and engage with the team that can repair it.

Many networking issues have similar symptoms, like "can't connect to server," at a high level. This level of detail is typically too generic to be useful in identifying the root cause, so you need to provide more diagnostic information. Network issues relate to connectivity, which always involves at least two specific points: source and destination. Always include information about these points when reporting network issues.

To structure your issue report,  use the conceptual tool of a packet flow diagram:

  • Describe the important hops that a packet takes along a path from source to destination, along with any significant transformations (e.g., NAT) along the way.
  • Start by identifying the affected network endpoints by Internet IP address or by RFC 1918 private address, plus an ASN for the network.
  • Note anything meaningful about the endpoints, such as who controls them and whether they are associated with a DNS hostname. 
  • Note any intermediate encapsulation and/or indirection. For example: VPN tunneling, proxies or NAT gateways.
  • Note any intermediate filtering, like firewalls, CDN or WAF.

Many problems that manifest as high latency or intermittent packet loss will require a path analysis and/or a packet capture for diagnosis. Path analysis is a list of all hops that packets traverse (for example, MTR or tcptraceroute). A packet capture (a.k.a. pcap, derived from the name of the library libpcap) is an observation of real network traffic. It's important to take a packet capture for both endpoints, at the same time, which can be tricky. Practice with the necessary tools (for example tcpdump or Wireshark) and make sure they are installed before you need them.

5. Escalate when appropriate
If circumstances change, you may need to escalate the urgency of an issue so it receives attention quickly. Take this step if business impact increases, if an issue is stuck without progress after a lot of back-and-forth with support, or if some other factor calls for quicker resolution.

The most explicit way to escalate an issue is to change the priority of the issue report (for example, from P3 to P2). Provide comments about why you need to escalate so support can respond appropriately.

6. Create a summary document for long-running or difficult issues
Issue state and relevant information change over time as new facts come to light and hypotheses are ruled out. In the meantime, new people join the investigation. Help communicate relevant, up-to-date information by collecting information in a summary document.

A good summary document has the following dimensions:

  • The latest state summarized at the top
  • Links to all relevant issue reports and internal tracking bugs
  • A list of hypotheses which are potentially true, and hypotheses that have been ruled out already. When you start investigating a particular hypothesis, note that you are doing so, and mention the tests or tools that you intend to use. Often, you can get good advice or prevent duplicate work.

SAMPLE summary document format:

<Current customer impact> <Working theory and actions being taken> <Next steps>

Customer impact has been mitigated and resolved. Our networking provider was throttling our traffic because we forgot to pay our bill last month. Next step is to be nicer to our finance team.

More than 100 customers are actively complaining about not being able to reach our service. Our networking provider is throttling customer traffic to one of our load balancers. The response team is actively working with our networking provider’s tier 1 support to understand why and how this happened.

We have now received 100 complaints from 50 customers from four different geos that they cannot consistently reach our API at Our engineers currently believe that an upstream networking issue is causing this. Next steps are to reach out to our networking provider to see if there are any upstream issues.

We have received five complaints from five customers that they are unable to reach Our engineers are looking into the issue.

Try to keep each issue report focused on a single issue. Don't reopen an issue report to bring up a new issue, even if it's related to the original issue. Do reference similar issues in your new report to help your provider recognize patterns from systemic root causes.

Keep your communication skills sharp

Communicating highly detailed technical information in a clear and actionable manner can be difficult. Doing so requires focus and specific skills. This task is particularly challenging in stressful situations, because our biological response to stress works against the need for clear cognitive reasoning. The following techniques help make communication easier for everyone.

Help reduce cognitive load by writing a detailed issue report
Many issue reports require the reader to make inferences or calculations. This introduces cognitive load, which decreases the mental energy available for solving the technical problem.

When writing an issue report, be as specific and detailed as possible. While this attention to detail requires more time on the part of the writer, consider that an issue report is written once but read many times by many people. People can solve the problem faster together when equipped with comprehensive information. Avoid acronyms and internal company code names. Also, be mindful of protecting customer privacy when disclosing any information to any third party.

Use narrative techniques
Once upon a time, in a land far, far away...

Humans are very good at absorbing information in the form of stories, so you can get your point across quite effectively this way. Start with the context: What was happening when you first observed the problem? What potential fixes did you try? Who are the characters involved, and why does the issue matter to them?
Include visuals Illustrate your issue report with any supporting images you have available, like text formatting or charts and screenshots.

Text formatting
Formatted text like log lines, code excerpts or MySQL results often become illegible when sent through plain-text emails. Add explicit markers (for example, <<<<<< at the end of the line) to help direct attention to important sections. You can use footnotes to point to long-form URLs, or use a URL shortener.

Use bullet points to format lists, and to call out important details like instance names. Use numbered lists to enumerate series of steps.

Charts are very useful for understanding time-series data. When you’re sending charts with an issue report, keep these best practices in mind:

  • Take a screenshot, including title and axis labels. For absolute values, specify units (requests per minute, errors per second, etc).
  • Annotate the screenshot with arrows or circles to call out important points.
  • Briefly describe what the chart is measuring.
  • Briefly describe how the chart normally looks.
  • In a few sentences, describe your interpretation of the chart and why it is relevant to the problem.

Avoid the following antipatterns:

  • The Y-axis represents a specific error (e.g., exceptions in my-handler) and has no clear relationship to the problem under investigation (e.g., high persistence-layer latency). To remedy this situation, explain why the graph is relevant to the problem.
  • The Y-axis is an absolute number (e.g., 1M per minute) that provides no context about the relative impact.
  • The X-axis doesn't have a time zone.
  • The Y-axis is not zero-based. This can make minor changes in the Y value seem very large.
  • Axis labels are missing or cut off.

Well-crafted issue reports, along with strong communication with your cloud provider, can speed the resolution process and time it takes. The cloud computing model has drastically changed the way that IT teams troubleshoot computer systems. Technical savvy is no longer the sole necessary skill set for effective troubleshooting--you must also be able to communicate clearly and efficiently with cloud providers. While the reality of your deployment may be unique, nuanced, and complex, these building blocks can help you navigate this territory.

Related content:
SLOs, SLIs, SLAs, oh my - CRE life lessons
SRE vs. DevOps: competing standards or close friends?
Introducing Google Customer Reliability Engineering

Special thanks to Ralph Pearson, J.C. van Winkel, John Lowry, Dermot Duffy and Dave Rensin

Why we’re committed to helping small businesses grow

I come from a family of small business owners. My grandparents owned a general store in Michigan, and in 1957 my parents followed suit and opened Robert L. Coe Plumbing and Heating in the town of Walled Lake. As the business grew, its success made it possible for my five siblings and me to go to college.

robert coe plumbing - cropped

Of course, my family and I aren't the only ones with good reason to be grateful for family businesses and small businesses. According to the U.S. Small Business Administration Office of Advocacy, there are nearly 30 million small businesses in the U.S., employing almost half of the private sector workforce. At Google, I lead a team that helps these businesses all over the world get online, reach customers and grow.

Google's digital tools help businesses tap into the potential of the web and reach customers at the very moment they’re looking to discover a product, learn about a new service or buy something. Every month, these search and advertising tools help drive 100 billion visits to business websites globally and create more than 3 billion direct connections like phone calls, online reservations and store visits between businesses and their customers.

Those connections have a huge impact on our economy. In 2017, Google’s search and advertising tools helped generate $283 billion in economic activity for American businesses, website publishers and nonprofits. Our annual Economic Impact Report, released today, features stories of businesses that are fueling that growth, creating job opportunities and transforming their communities—like Carousel Designs, a family-owned nursery design company in Georgia that was able to revitalize its business through online ads and grow from four employees to 70 employees.

For any small business, success hinges on the ability to reach and connect with customers. For my parents, this meant advertising on matchbooks, on company trucks and in the Yellow Pages (which we used to flip through to see how my family’s ad appeared). For today’s small businesses it usually starts with a website and online ads.

However, only 51 percent of businesses in the U.S. have a website. Outside of the U.S., that figure is even lower; 29 percent in Brazil, 26 percent in Mexico and 26 percent in Argentina. The reality is that without using the web to find and connect with customers, small business owners are missing out on a big opportunity.

That’s why our initiatives are built to help even the smallest businesses take steps toward benefiting from being online. For starters, the Google My Business platform is a free and easy way for businesses to show up in Google Search and Google Maps and even quickly create a website.  More than 3 million small business owners around the globe have already used the website builder tool and taken fundamental steps toward connecting with their customers. But we don’t stop there—because as a business evolves, so does the need to maintain or grow its presence through advertising online. We work to provide tools, like AdWords Express, that make advertising online a little bit easier and a little bit more accessible for people.

Small business owners have a special dedication and passion for what they do, as well as a deep understanding for the economics of their business. We’ve seen what small business owners can accomplish when they combine those strengths with the power of digital tools. That’s why we're more committed than ever to helping small businesses succeed. Because when they do, it creates a positive ripple effect that helps all of us—whether that’s creating more jobs in our communities or helping a child (or a whole generation of them!) go to college and pursue their dreams.

The web is working for American businesses: stories of economic impact across the U.S.

The web is making it possible for U.S. businesses to grow and succeed. Digital tools create significant opportunities for small- and medium-sized businesses. In fact, digital small businesses are growing faster, reaching more customers and were nearly three times as likely to have created jobs over the previous year.

In 2017, Google’s search and advertising tools helped provide $283 billion of economic activity for businesses, website publishers, and nonprofits across the U.S. Our 2017 Economic Impact Report shares that economic impact state-by-state as well as stories of local businesses that are fueling that growth, creating jobs, and transforming their communities. Here are some of those stories:  

American Hats

After more than three decades of working as a social worker, Georgiette Morgan-Thomas put off her plans for retirement and purchased the S & S Hat Company in Philadelphia, PA when it announced its closure in 2015. She assured the employees that she would fight to keep it alive. Together with her son, Robert, she started American Hats in 2016, and turned to Google’s search and advertising tools to help reinvigorate the business, which was good news for the workers. “People are just flying through the door now, and we’re even shipping across the country,” says Georgiette. “Our employees have been with this factory for 20 to 30 years. They're the original team that made S & S Hat so outstanding in the first place.”

American Hats in Philadelphia, PA

Merz Apothecary

Merz Apothecary in Chicago, IL has been home to a rich collection of unique, hard-to-find goods from all over the world since 1875. “We’re not your traditional pharmacy,” says co-owner Anthony Qaiyum. “We carry everything from natural remedies to grooming products to home fragrances. People always say: ‘If you can’t find it at Merz, you won’t find it anywhere else!’”

Two decades after they took their first online order in 1997, Merz Apothecary has a thriving e-commerce operation serving 35,000 customers a year. The company shares product reviews on their YouTube channel, and special sales via Posts on their Google My Business listings. And with Google Search and AdWords, they reach customers anywhere in the world. Since Anthony joined the family business full-time in 2000, they went from 12 employees to 60. “As the owner of Merz, I feel like I’m the caretaker of a piece of history. And I take that duty very seriously,” says Anthony. “I want to leave this business better and stronger than it was when I joined.”


Merz Apothecary in Chicago, IL

Carousel Designs

Jonathan Hartley’s parents founded Carousel Designs in Douglasville, GA in 1988, manufacturing baby bedding for other retailers. As the cut and sew industry moved overseas, Jonathan realized that the family business would need to innovate in order to stay competitive. After graduating from the U.S. Military Academy at West Point, he and his former classmate, Allan Sicat, bought the company in 2007. They began marketing their products directly to consumers, using AdWords to connect with customers and Google Analytics to improve their online shopping experience. Carousel Designs now offers thousands of nursery decor options to customers across the U.S. and Canada, and employs nearly 70 people. “It’s something we’re very proud of,” Allan adds. “Not only being made in the USA, but also being able to create these opportunities right here in Douglasville.”


Carousel Designs in Douglasville, GA

As you can see from these stories, the web is working for American businesses large and small. Check out more stories of businesses growing with the web.

Introducing Neighbourly: the app for trusted, neighbourhood answers
The Beta app is available in Mumbai starting today

Life happens close to home. In the course of a day, local questions come up all the time: Know a good pediatrician for my daughter? Which film festivals in our locality have free entry for kids? Any reliable mechanics nearby?

In cities across India, it’s getting harder to get good answers to these questions because cities keep changing, including the people who live there. People nearby often have the exact information you need, but they’re too busy these days to stop and chat on the street or in the park. And group chats keep getting bigger and noisier — and often fill up with those "Good Morning" messages.

And yet we need answers. That's why we built Neighbourly, a neighbourhood app that helps you ask your neighbours questions, share local expertise, and keep up with your neighbourhood. With Neighbourly, your questions get routed to the relevant neighbours instantly, and then they can write back with the most up-to-date and accurate info.
Neighbourly is a human, helpful and local way to ask and answer neighbourhood questions

Neighbourly helps you:

  • Ask your neighbours questions. Ask a question for your family, and it's routed to the neighbourhood experts who can help. Using Google's voice recognition, you can speak your question or answer into the app, just like talking to a neighbour. It works in English and eight Indian languages.
  • Share your local expertise. Easily swipe left and right through the questions and give answers. The more you answer, the more you’re recognized for your local expertise. Plus, every time you answer, that information directly helps one of your neighbours and makes your entire neighbourhood more informed.
  • Keep up with your neighbourhood safely. Browse, ask and answer questions without sharing all of your personal information. Personal details like your phone number, full name and other contact information are kept private when you use Neighbourly. And every member that joins Neighbourly makes the Neighbourly Promise so we can all work together to keep the app’s community safe and helpful.

Starting today, we’re making a Beta version of Neighbourly available on the Google Play Store for all smartphones in Mumbai running Android 4.3 (Jelly Bean) and higher. If you live outside Mumbai, join our waitlist and invite your neighbours. We’ll be looking to expand to other neighbourhoods across India that are excited to try Neighbourly.

Today is the first step for Neighbourly, and we want your feedback. We hope Neighbourly, like Google Search, can be another way you find the information you need, and, in the process, make neighbourhoods everywhere feel even more neighbourly.

Posted by Caesar Sengupta, VP, Next Billion Users Team

Announcing the Google Search Conference 2018

Regional language Internet usage is growing at an unprecedented pace in India and generating great demand for local language content. While many publishers have focused on Hindi, there is now growing demand for other regional languages. We are delighted to share that we are bringing back the Google Search conference with an objective to help Indian language publishers make the most of the opportunity and bridge the Indian language content gap that exists on the web today.

We will be hosting conferences across 11 cities in India to help local language publishers and webmasters better understand how they can make their content easily accessible to the growing number of Internet users across India. In addition to Hindi, this year we will cover four other Indian languages such as Tamil, Telugu, Marathi and Bengali.

Conference cities and dates -

  • Gurgaon on June 20 (Wednesday)
  • Pune on June 22 (Friday)
  • Indore on July 2 (Monday)
  • Patna on July 4 (Wednesday)
  • Lucknow on July 6 (Friday)
  • Hyderabad on July 16 (Monday)
  • Visakhapatnam on July 18 (Wednesday)
  • Kolkata on July 20 (Friday)
  • Coimbatore on July 30 (Monday)
  • Chennai on Aug 1 (Wednesday)
  • Bengaluru on Aug 3 (Friday) - this event is focused on women webmasters

At each day-long conference, members of the Google Webmaster Outreach team will talk about a host of topics including ‘How Search Works’, ‘Tips for better visibility of Indian language websites in Google Search’, ‘Best practices for mobile-friendly websites’, ‘Google’s Search quality guidelines’ and many other important topics. We are also including a  session from the Google AdSense team to elucidate their policies and share how to avoid mistakes while running AdSense on your sites.

Please note that this is an invite-only event and filling out the form does not guarantee your spot at the conference. Please register your interest here! (The registration for the aforementioned cities closes 10 days before the listed date). Once selected, you will receive an invitation email from us confirming the venue details.

Posted by Vandana Bharvani, Director, Research & Outreach - Trust & Safety

Making it even easier to join meetings from more third-party clients

In August 2017, we announced a feature that made it even easier to join a Google Calendar meeting using a CalDAV-based third-party calendaring client. We’ve heard from you that this functionality is helpful and you’d like to see it expanded to more calendaring clients. That’s why now, whenever someone uses G Suite Sync for Microsoft Outlook, Google Sync, or a third-party Android client that relies on Calendar Provider to access an event on Google Calendar, we’ll add automatically generated text to the event’s description. This text will include instructions on how to join the event through Meet or Hangouts.

With this launch, people using Google Calendar clients that don’t display meeting instructions natively (e.g., Samsung S Planner) no longer need to go to Google Calendar on the web to join their meetings.

Launch Details
Release track:
Launching to Rapid Release with Scheduled Release coming two weeks later

Available to all G Suite editions

Rollout pace:
Full rollout (1–3 days for feature visibility)

Impact:All end users using Suite Sync for Microsoft Outlook, Google Sync or a third-party Android client that relies on Calendar Provider

Change management suggested/FYI

Launch release calendar
Launch detail categories
Get these product update alerts by email
Subscribe to the RSS feed of these updates

Celebrating extraordinary Australian AI stories

For over a decade, we’ve been working on artificial intelligence to make more of the world’s information accessible and useful. Today, AI is infused in almost all our products – helping you to snap the perfect selfie with Portrait Mode on Pixel 2, breaking down language barriers in Google Translate, and making it easier to respond to an email with Smart Reply in Gmail. We’ve even taken AI into the kitchen, teaming up with a bakery to whip up the world’s first smart cookie.

But the impact of AI goes beyond making products more useful (or tasty). We’re starting to see how AI can be a positive force for society––from high school students who built a machine learning tool to detect diseases on plants and skin, to conservationists preventing illegal logging in the Amazon rainforest with AI – and researchers partnering with astronomers hunting new planets with machine learning. The potential of AI to solve complex, real-world problems is huge. To help more people tackle challenges with AI, we’ve open-sourced machine learning tools like TensorFlow, we help others innovate with Cloud AI – and collaborate with researchers around the globe.

And here in Australia, researchers, developers and businesses are using these AI tools to solve difficult problems in the fields of heath, conservation, linguistics and more. Today, we celebrated some of these stories with an event in our Sydney office, to show how AI is driving impact in extraordinary, unexpected and tangible ways––here and now.

Here’s a snapshot of some Aussie AI-powered products and projects in a range of fields:

Saving dugongs with AI
Credit: Ahmed M. Shawky

Dugongs are the gentle giants of the sea and despite their size, are hard to keep track of. This has presented a challenge for conservation researchers working to save this endangered species.  For decades, scientists had to spend days peering out of small planes to count populations, which was expensive, time consuming and often hazardous. Researchers then analysed imagery manually, zooming in to count dugongs one-by-one.

Dr. Amanda Hodgson of Murdoch University and Dr. Frederic Maire of Queensland University of Technology knew there must be a better way. In 2010, they began testing drones, which take aerial photography of the ocean––and in 2014, they applied the magic of machine learning in their quest to make the processing of images from drones faster and cheaper, and therefore make drone surveys a realistic option. Using TensorFlow, Google’s free open source machine learning platform, the team built a detector that could learn to find dugongs in these photos automatically.  

So far, the team have processed 37k+ images––identifying 70% of the sea cows they’d found manually in images. This analysis took 18 hours to complete, compared to the 377 hours required for manual analysis. Hodgson and Frederic have now integrated this detection software with mapping software to plot all sightings, giving them richer data about the volume and locations of dugongs.

Preserving precious languages
While there are 6,000+ languages in the world, only 50% of web content is in English. AI can help us make this content accessible, break down language barriers and even preserve endangered languages. Since 2012, Google's language technology teams have been using neural networks to make the world’s diverse language content universally accessible and useful.

Professor Janet Wiles and Ben Foley, researchers with the ARC Centre of Excellence for the Dynamics of Language (CoEDL) are working to transcribe and preserve endangered languages. There are over 300 Indigenous languages in Australia––which can be as distinct as Japanese is to German. Indigenous languages are also inextricably connected to the land, imbued with history and sacred songlines––passed down through oral tradition.

CoEDL conducting traditional methods of fieldwork

With many indigenous languages endangered, research and transcription is both time sensitive and intensive. CoEDL has fieldworkers working with ~130 languages, recording mountains of data (almost 50K hours of language audio in archives), which could take 1.9M hours to transcribe using traditional methods. Recognising the importance and the sheer enormity of the work, Wiles and Foley realized AI could help provide a new solution to harness the contributions of community members and linguists, while protecting the integrity of this precious language data.

CoEDL and Google teams building language models at a recent workshop

In 2016, Wiles and Foley looked to Google’s open-source AI technology to build unique models for several Indigenous languages––allowing for faster transcription and a bespoke solution. While this project is still in its early stages, Google and CoEDL are delighted to announce this partnership to implement TensorFlow and Kaldi to transcribe indigenous languages. So far, we’ve co-hosted workshops with 35 linguists, and have built initial models for 12 Indigenous languages including Bininj Kunwok, Kriol, Mangarayi, Nakkara, Pitjantjatjara, Warlpiri, Wubuy – as well as indigenous languages in regions surrounding Australia, such as Abui (spoken in Indonesia) and Cook Islands Maori.

CoEDL aim to train more language workers to contribute to the models, and build an even simpler interface. Long-term, the team has a dream to integrate recognition and synthesis systems into their social robot Opie, designed with the Ngukurr Language Center to promote community engagement and the revitalisation of endangered languages.

Enhancing healthcare with AI

There’s a huge opportunity for AI to help solve numerous difficult problems, and in healthcare we’re already seeing some really encouraging applications that could benefit billions of people.  Working closely with clinicians and medical providers, we’re developing tools that we hope will improve the availability and accuracy of medical services across a range of conditions, from diabetic eye disease, to cardiovascular health, and cancer.

In early 2017 we partnered with Dr Elliot Smith, of Brisbane-based medical data specialist Maxwell Plus, to combine deep learning with medical imaging to diagnose prostate cancer in a faster, more affordable and accurate way.  Dr Smith, an expert in magnetic resonance imaging (MRI) systems with a Ph.D. in Electrical Engineering, was troubled that highly trained diagnostics are a scarce resource and unevenly distributed in Australia. Moreover, prostate cancer diagnostic methods can take up to seven days to process results. Smith felt compelled to find a scalable solution and make this clinical brainpower available to all doctors, to offer patients better care.

Dr Smith leveraged Google Cloud AI to ‘train’ a system to analyse hundreds of thousands of prostate cancer images using AI. This delivered results to clinicians in 10-15 minutes rather than 2-7 days by traditional diagnostic methods. Maxwell Plus has since expanded to cover breast and lung cancer diagnostics – and has a goal to process 150,000 cases by the end of 2018.

Maxwell Plus’ interface running cancer diagnostics, powered by Google Cloud Platform

Cherishing memories, art and culture

Advances in computer vision and mobile photography allow you to search, stylise and share your photo––and learn about the world around you.

Three years ago, we introduced Google Photos as a home for all of your pictures and videos, organized and brought to life. One AI powered feature which will be rolling out soon is Colour Pop, which detects the subject of your photo through machine learning and leaves it in colour–– while the background is set to black and white.

We’ve also seen breakthroughs in computational photography when AI, software and hardware come together. HDR+ (which runs on all recent Pixel and Nexus phones) produces photos and videos with low noise and sharp details, even in dim lighting. To see this in action, here’s a video by Aussie surf photography leader Aquabumps, showcasing HDR+ quality in video:

The Pixel 2 also contains a specialized neural network to produce portrait mode, which researchers trained on almost a million images. By using machine learning, the device can make predictions about what should stay sharp in the photo and create a mask around it, producing professional-looking shallow depth-of-field image. Here are a few portrait mode snaps taken in Australia by photographers @karin_samsovona and @samscrim.

Australian AI stories offer a glimpse of the potential of AI to improve people’s lives––giving you the tools to capture your most precious moments, supporting doctors to serve their patients, helping animal populations grow, and preserving languages to live on. It is a privilege to partner with so many brilliant minds and creative thinkers to discover new applications of AI, and uncover new avenues to tackle some of our most pressing social issues.   

Dev Channel Update for Desktop

The dev channel has been updated to 68.0.3440.7 for Windows, Mac and Linux.

A partial list of changes is available in the log. Interested in switching release channels? Find out how. If you find a new issue, please let us know by filing a bug. The community help forum is also a great place to reach out for help or learn about common issues.

Abdul Syed
Google Chrome

Creating AR Experiences for I/O: Our Process

Posted by Karin Levi, Product Marketing, ARCore

A few weeks ago at Google I/O we released a major update to ARCore, Google's AR development platform. We added new APIs like Cloud Anchors, that enable multi-user, collaborative AR experiences and Augmented Images that enable activation of 2D images into 3D objects. All of these updates are going to change the way we use AR today and enable developers to create richer, more immersive AR apps.

With these new capabilities, we decided to put our platform to the test. So we built real experiences to showcase how these all come to life. All demos were presented at the I/O AR & VR sandbox area. We open sourced them to make sure you can see how simple it is to build these experiences. We're pretty happy with how they turned out and would love to share with you some learning and insights from behind the scenes.

Light Board - Multiplayer game

Light Board is an AR multiplayer tabletop game where two players on floating game boards launch colored projectiles at each other.

While building Light Board it was important for us to keep in mind who the end users are. We wanted it to be a simple/fun game for developers to try out while visiting the I/O sandbox. The developers would only have a couple minutes to play while passing through, so it needed to allow players (even non-gamers) to pick it up and play with very little setup.

The artwork for Light Board was a major focus. Our mission for the look of the game was to align with the design and decor of I/O 2018. This way, our app would feel like an extension of everything the attendees saw around them. As a result, our design philosophy had 3 goals; bright accent colors, simple graphic shapes and natural physical materials.

Left: Design for AR/VR Sandbox at I/O 2018. Right: Key art for Light Board game boards

The artwork was created in Maya and Cinema 4D. We created physically based materials for our models using Substance Painter. Just as continuous iteration is crucial for engineering, it is also important when creating art assets. With that in mind, we kept careful track of our content pipeline, even for this relatively simple project. This allowed us to quickly try out different looks and board styles before settling on our final design.

On the engineering front we selected the Unity game engine as our dev environment. Unity gives us a couple of important advantages. First, it is easy to get great looking 3D graphics up and running right away. Second, the engine component is already complete, so we could immediately start iterating on gameplay code. As with the artwork, this allowed us to test gameplay options before we made a final decision. Additionally, Unity gave us support for both Android and iOS with only a little extra work.

To handle the multiplayer aspect we used Firebase Realtime Database. We were concerned with network performance at the event, and felt that the persistent nature of a database would make it more tolerant of poor networks. As it turned out, it worked very well and we got the ability to quit and rejoin games for free!

We had a lot of fun building Light Board and we hope people can use it as an example of how easy it can be to not only build AR apps, but to use really cool features like Cloud Anchors. Please check out our open source repo and give Light Board a try!

Just a line - Draw with your friends

In March, we released Just a Line, an Android app that lets you draw in the air with your phone. It's a simple experiment meant to showcase the power of ARCore. At Google I/O, we added Cloud Anchors to the app so that two people can draw at once in the same space, even if one of them is using Android and the other iOS.

Both apps were built natively: The Android version was written in Android Studio, and the iOS version was built in xCode. ARCore's Cloud Anchors enable Just a Line to pair two phones, allowing users to draw simultaneously in a shared space. Pairing works across Android and iOS devices, and drawings are synchronized live through a Firebase Realtime Database. You can find the open-source code for iOS here and for Android here.

Illusive Images - Art exhibition comes to life

"Illusive Images" demo is an augmented gallery consisting of 3 artworks, each exploring a different augmented image use case and user experience. As one walks from side to side, around the object, or gazes in a specific direction, 2D artworks are married with 3D, inviting the viewer to enter into the space of the artwork spanning well beyond the physical frame.

Due to the visual design nature of our augmented images, we experimented a lot with creating databases with varying degrees of features. In order to get the best results, we iterated quickly by resizing the canvas for the artwork. We also moved and stretched the brightness and contrast levels. These variations helped to achieve the most optimal image without compromising design intent.

The app was built in Unity with ARCore, with the majority of assets created in Cinema 4D. Mograph animations were imported into Unity as an fbx, and driven entirely by the position of the user in relation to the artwork. An example project can be found here.

To make your development experience easier, we open sourced all the demos our team built. We hope you find this useful! You can also visit our website to learn more and start building AR experiences today.