Monthly Archives: January 2017

Dev Channel Update for Desktop

The dev channel has been updated to 57.0.2987.19 for Windows, Mac and Linux.


A partial list of changes is available in the log. Interested in switching release channels? Find out how. If you find a new issue, please let us know by filing a bug. The community help forum is also a great place to reach out for help or learn about common issues.


Krishna Govind
Google Chrome

New resources for building inclusive tech hubs

Posted by Amy Schapiro and the Women Techmakers team

For the tech industry to thrive and create groundbreaking technology that supports the global ecosystem, it is criticalto increase the diversity and inclusion of communities that make the technology. To support this global network of tech hubs - incubators, community organizations, accelerators and coworking spaces - Women Techmakers partnered with Change Catalyst to develop an in-depth video series and set of guides on how to build inclusive technology hubs.


Watch the videos on the Women Techmakers YouTube channel, and access the how-to guides on the Change Catalyst site [via this link].


For more information about Women Techmakers, Google's global program supporting women in technology, and to join the Membership program, visit womentechmakers.com.

A remedy for your health-related questions: Google Search launches health info in the Knowledge Graph

Think of the last time you searched on Google for health information. Maybe you heard a news story about gluten-free diets and Googled, "What is celiac disease?." Maybe you were planning a trip to Bali, so you looked up “Zika virus” so you could take precaution. Or perhaps you were worried about a loved one, and searched to better understand their condition and prognosis.

Starting today in Australia, Google search results will show information for over 900 commonly searched for health conditions. The new health feature will include an outline of the condition, symptoms, diagnosis, and prevalence according to age at the top of search results. For some conditions you’ll also see high-quality illustrations from licensed medical illustrators. Once you get this basic info from Google, you should find it easier to do more research on other sites around the web, or know what questions to ask your doctor.

This feature has been developed as more people turn to Google to help with their health queries. In fact, 1 in 20 Google searches are for health-related information. That’s why we’ve surfaced trusted, quality health information right in Google Search. We hope this will help people find the information they need more quickly and easily.
We worked with a team of medical doctors (led by our own Dr. Kapil Parakh, M.D., MPH, Ph.D.) to carefully compile, curate, and review this information. All of the gathered facts represent real-life clinical knowledge from these doctors and high-quality medical sources across the web. Each fact has been checked by a panel of at least ten medical doctors at Google and the Mayo Clinic for accuracy.

That doesn’t mean these search results are intended as medical advice. We know that cases can vary in severity from person to person, and that there are bound to be exceptions. What we present is intended for informational purposes only — and you should always consult a healthcare professional if you have a medical concern.

We hope this information will empower you in your health decisions – so the next time you need information on measles or treatments for tennis elbow, Google will be a better place to start.

Discover the new AdWords experience

In 2016 we introduced a new AdWords experience and even shared an early look at it in action. From the beginning, we asked ourselves, “How can we make AdWords as relevant for the next 15 years as the first 15?” And during this multi-year journey, we met with many of you -- across industries and around the world, from power users to beginners -- to learn what's working, and what could be improved.

We learned a lot, and it’s helped shape what we’ve built. First, AdWords should focus on your business, and not around features. The data you care about should be available at your fingertips. And finally, the tools you use to get things done should be simple yet powerful.

This comes together in the new experience in many ways. For example, the new Overview page surfaces relevant insights about your account and helps you visualize them so you can take action. We’ve removed the clutter and made navigation a breeze so you can do more in less time. And in the coming months, we’re introducing new tools, unique to this experience, that will help you reach customers in new ways.

I recently sat down with a few advertisers who told me the new experience "is faster," “more intuitive,” and that “it’s easier to get things done.” It’s so inspiring to hear this feedback because we’re aiming to do exactly that -- build a platform that’s easy to use and helps you reach your unique business goals.

We’ve made so much progress since our announcement last March, and I’m happy to share that we’re rolling out the new AdWords experience to more of you within the next few months. Some of you will automatically see it when you sign in to AdWords, while others will see a message to try it out.* Though not everyone will have access to the new experience at the same time, we encourage those who do to explore it, discover new insights, and provide feedback as we continue to refine and roll it out to more advertisers.

Examples of Overview cards**


To get started, I recommend taking the guided tour to familiarize yourself with the new experience. The AdWords Help Center is also being updated to provide additional support as you get acquainted with all the new features.

We look forward to continuing this journey in 2017 and can’t wait to share the new AdWords experience with more of you.



* Invites are sent based on a number of factors, including the features you use
** Campaign performance, Auction Insights, and Locations cards are coming soon

Source: Inside AdWords


Announcing new enterprise-grade controls and visibility in G Suite

(Cross-posted from The Keyword)

Posted by: Reena Nadkarni, Product Manager, G Suite

Scale. Speed. Lower total cost of ownership. The cloud offers enterprises significant benefits and gives their teams a freedom that’s as big as their ambitions. G Suite supports such businesses around the world with our unified suite of intelligent apps. Today, more than 3 million paying businesses rely on G Suite to make working together easier, including Whirlpool, PWC, and Woolworths.

Having greater control and visibility when protecting sensitive assets, however, should also be a top concern in today’s world. That’s why starting today, we’re giving customers the critical control and visibility they expect (and their CTOs and regulators often require) in G Suite with:

  • More powerful access control for administrators with Security Key enforcement 
  • More data control with Data Loss Prevention (DLP) for Google Drive, DLP for Gmail, and S/MIME for Gmail 
  • More analytics and insights by connecting BigQuery with Gmail 


Manage who can access your company’s devices and systems 


Security Key enforcement: In 2011, we introduced 2-step verification for added protection when users log into their Google accounts. With 2-step verification, admins can opt for their company to use Security Keys, which are both easier for users and more secure against common (phishing) attacks than other methods. Security Keys plug directly into a laptop or pair with a mobile device using Bluetooth or NFC, instead of receiving a code via text. Today, we’re adding the ability for admins to enforce this measure by restricting login solely to users with a Security Key. Admins will also be able to manage the deployment of Security Keys and view usage reports.


--------------------------------------------

See how Stripe enhances account security with G Suite  
Stripe, a software platform for running an internet business, processes billions of dollars a year for hundreds of thousands of companies around the world. Everyday, Stripe employees rely on G Suite to communicate and collaborate quickly, effectively and securely. Security Key enforcement provides an added layer of protection for Stripe, with advanced authentication controls to combat phishing attempts and unauthorized email access.
“Businesses that run on Stripe trust us with some of their most sensitive information, and we take the security of that information extremely seriously. We’ve found Security Keys to be ideal second factors because they balance ease-of-use with increased account security. G Suite Security Key enforcement gives us a way to easily enforce secure logins for all employees, across both their desktop and now mobile devices,” says Bryan Berg, security team member at Stripe.

--------------------------------------------


Control how data is shared beyond your company 

  • Data loss prevention (DLP) for Google Drive: In 2015, we launched DLP for Gmail with easy-to-set-up rules, OCR image scan recognition and advanced customization. Today, we’re extending DLP to Google Drive, making it even easier for admins to secure sensitive data, control what content can be stored and protect users from accidentally sharing confidential information externally. G Suite’s DLP protection goes beyond standard DLP with easy-to-configure rules and OCR recognition of content stored in images so admins can easily enforce policies and control how data is shared.  


  • S/MIME for Gmail: When it comes to Gmail security, there are dozens of security measures running behind the scenes to keep your emails safe, and we support industry-standard authentication to help combat email spoofing. Unfortunately, other email services that you exchange emails with might not take similar measures to protect your data. That’s why we’re offering additional protection by giving you the option to bring your own certificates for S/MIME encryption. Administrators will be able to enforce S/MIME usage and can set DLP rules based on specific organizational unit needs. 

Understand employee behavior and activity

  • Gmail logs + BigQuery integration: Gmail logs contain valuable information that can help administrators diagnose issues or unlock insights. Today, we are making it easier to analyze Gmail logs with a preconfigured BigQuery integration so administrators can run sophisticated, high-performing custom queries, analyze their data and build custom dashboards. 


  • Third-party email archiving: Google Vault helps you easily archive Gmail content while also complying with retention for legal holds and audit reporting. That said, some organizations use other third-party SMTP archiving solutions like HP Autonomy or Veritas. Today, we’re making it simple for you to integrate third-party archiving solutions of your choice with Gmail.

All of these features are part of our new G Suite Enterprise edition. While the features are designed to meet the demands of G Suite's large customers, organizations of any size can take advantage of these critical controls, seamless integrations, and intelligent apps. See if G Suite Enterprise edition is right for you.

Launch Details
Release track:
All features available to G Suite customers on both Rapid and Scheduled release who upgrade to G Suite Enterprise edition. Note that G Suite for Education customers are not eligible to upgrade to G Suite Enterprise. G Suite for Education customers interested in these new features should complete this form.

Editions:
All features available to G Suite Enterprise edition customers only

Impact:
Admins only

Action:
Admin action suggested/FYI

More Information
Data loss prevention (DLP) for Drive: 
Scan and protect Drive files using data loss prevention rules

Data loss prevention (DLP) for Gmail: 
Scan your email traffic using data loss prevention

Security Key enforcement: 
2-step verification enforcement

Hosted S/MIME for Gmail: 
Enable hosted S/MIME for enhanced message security

Gmail logs + BigQuery integration: 
Enable email logs in BigQuery

Third-party archiving tools for Gmail: 
Integrate Gmail with a third-party archiving solution


Launch release calendar
Launch detail categories
Get these product update alerts by email
Subscribe to the RSS feed of these updates

Announcing new enterprise-grade controls and visibility in G Suite

Scale. Speed. Lower total cost of ownership. The cloud offers enterprises significant benefits and gives their teams a freedom that’s as big as their ambitions. G Suite supports such businesses around the world with our unified suite of intelligent apps. Today, more than 3 million paying businesses rely on G Suite to make working together easier, including Whirlpool, PWC, and Woolworths.

Having greater control and visibility when protecting sensitive assets, however, should also be a top concern in today’s world. That’s why starting today, we’re giving customers the critical control and visibility they expect (and their CTOs and regulators often require) in G Suite with:

  • More powerful access control for administrators with security key enforcement
  • More data control with Data Loss Prevention (DLP) for Google Drive, DLP for Gmail, and S/MIME for Gmail
  • More analytics and insights by connecting BigQuery with Gmail

Manage who can access your company’s devices and systems

  • Security key enforcement:  In 2011, we introduced 2-step verification for added protection when users log into their Google accounts. With 2-step verification, admins can opt for their company to use Security Keys, which are both easier for users and more secure against common (phishing) attacks than other methods. Security Keys plug directly into a laptop or pair with a mobile device using Bluetooth or NFC, instead of receiving a code via text. Today, we’re adding the ability for admins to enforce this measure by restricting login solely to users with a Security Key. Admins will also be able to manage the deployment of Security Keys and view usage reports.

See how stripe enhances account security with G Suite

Stripe, a software platform for running an internet business, processes billions of dollars a year for hundreds of thousands of companies around the world. Everyday, Stripe employees rely on G Suite to communicate and collaborate quickly, effectively and securely. Security Key enforcement provides an added layer of protections for Stripe, with advanced authentication controls to combat phishing attempts and unauthorized email access. 

"Businesses that run on Stripe trust us with some of their most sensitive information, and we take the security of that information extremely seriously. We've found Security Keys to be ideal second factors because they balance ease-of-use with increased account security. G Suite Security Key enforcement gives us a way to easily enforce secure logins for all employees, across both their desktop and now mobile devices, " says Bryan Berg, security team member at Stripe.

Control how data is shared beyond your company

  • Data loss prevention (DLP) for Google Drive: In 2015, we launched DLP for Gmail with easy-to-set-up rules, OCR image scan recognition and advanced customization. Today, we’re extending DLP to Google Drive, making it even easier for admins to secure sensitive data, control what content can be stored and protect users from accidentally sharing confidential information externally. G Suite’s DLP protection goes beyond standard DLP with easy-to-configure rules and OCR recognition of content stored in images so admins can easily enforce policies and control how data is shared.

GIf 1

  • S/MIME for Gmail: When it comes to Gmail security, there are dozens of security measures running behind the scenes to keep your emails safe, and we support industry-standard authentication to help combat email spoofing. Unfortunately, other email services that you exchange emails with might not take similar measures to protect your data. That’s why we’re offering additional protection by giving you the option to bring your own certificates for S/MIME encryption. Administrators will be able to enforce S/MIME usage and can set DLP rules based on specific organizational unit needs.

Understand employee behavior and activity

  • Gmail logs + BigQuery integration: Gmail logs contain valuable information that can help administrators diagnose issues or unlock insights. Today, we are making it easier to analyze Gmail logs with a preconfigured BigQuery integration so administrators can run sophisticated, high-performing custom queries, analyze their data and build custom dashboards.

gif 2

  • Third-party email archiving: Google Vault helps you easily archive Gmail content while also complying with retention for legal holds and audit reporting. That said, some organizations use other third-party SMTP archiving solutions like HP Autonomy or Veritas. Today, we’re making it simple for you to integrate third-party archiving solutions of your choice with Gmail.

Part of the new G Suite Enterprise edition, today’s announcement is designed to provide the peace-of-mind our enterprise customers demand and deserve.

See how your business can innovate faster — and now even more securely — with G Suite by Google Cloud.

Source: Drive


Announcing new enterprise-grade controls and visibility in G Suite

Scale. Speed. Lower total cost of ownership. The cloud offers enterprises significant benefits and gives their teams a freedom that’s as big as their ambitions. G Suite supports such businesses around the world with our unified suite of intelligent apps. Today, more than 3 million paying businesses rely on G Suite to make working together easier, including Whirlpool, PWC, and Woolworths.

Having greater control and visibility when protecting sensitive assets, however, should also be a top concern in today’s world. That’s why starting today, we’re giving customers the critical control and visibility they expect (and their CTOs and regulators often require) in G Suite with:

  • More powerful access control for administrators with security key enforcement
  • More data control with Data Loss Prevention (DLP) for Google Drive, DLP for Gmail, and S/MIME for Gmail
  • More analytics and insights by connecting BigQuery with Gmail

Manage who can access your company’s devices and systems

  • Security key enforcement:  In 2011, we introduced 2-step verification for added protection when users log into their Google accounts. With 2-step verification, admins can opt for their company to use Security Keys, which are both easier for users and more secure against common (phishing) attacks than other methods. Security Keys plug directly into a laptop or pair with a mobile device using Bluetooth or NFC, instead of receiving a code via text. Today, we’re adding the ability for admins to enforce this measure by restricting login solely to users with a Security Key. Admins will also be able to manage the deployment of Security Keys and view usage reports.

See how stripe enhances account security with G Suite

Stripe, a software platform for running an internet business, processes billions of dollars a year for hundreds of thousands of companies around the world. Everyday, Stripe employees rely on G Suite to communicate and collaborate quickly, effectively and securely. Security Key enforcement provides an added layer of protections for Stripe, with advanced authentication controls to combat phishing attempts and unauthorized email access. 

"Businesses that run on Stripe trust us with some of their most sensitive information, and we take the security of that information extremely seriously. We've found Security Keys to be ideal second factors because they balance ease-of-use with increased account security. G Suite Security Key enforcement gives us a way to easily enforce secure logins for all employees, across both their desktop and now mobile devices, " says Bryan Berg, security team member at Stripe.

Control how data is shared beyond your company

  • Data loss prevention (DLP) for Google Drive: In 2015, we launched DLP for Gmail with easy-to-set-up rules, OCR image scan recognition and advanced customization. Today, we’re extending DLP to Google Drive, making it even easier for admins to secure sensitive data, control what content can be stored and protect users from accidentally sharing confidential information externally. G Suite’s DLP protection goes beyond standard DLP with easy-to-configure rules and OCR recognition of content stored in images so admins can easily enforce policies and control how data is shared.

GIf 1

  • S/MIME for Gmail: When it comes to Gmail security, there are dozens of security measures running behind the scenes to keep your emails safe, and we support industry-standard authentication to help combat email spoofing. Unfortunately, other email services that you exchange emails with might not take similar measures to protect your data. That’s why we’re offering additional protection by giving you the option to bring your own certificates for S/MIME encryption. Administrators will be able to enforce S/MIME usage and can set DLP rules based on specific organizational unit needs.

Understand employee behavior and activity

  • Gmail logs + BigQuery integration: Gmail logs contain valuable information that can help administrators diagnose issues or unlock insights. Today, we are making it easier to analyze Gmail logs with a preconfigured BigQuery integration so administrators can run sophisticated, high-performing custom queries, analyze their data and build custom dashboards.

gif 2

  • Third-party email archiving: Google Vault helps you easily archive Gmail content while also complying with retention for legal holds and audit reporting. That said, some organizations use other third-party SMTP archiving solutions like HP Autonomy or Veritas. Today, we’re making it simple for you to integrate third-party archiving solutions of your choice with Gmail.

Part of the new G Suite Enterprise edition, today’s announcement is designed to provide the peace-of-mind our enterprise customers demand and deserve.

See how your business can innovate faster — and now even more securely — with G Suite by Google Cloud.

Source: Gmail Blog


Announcing new enterprise-grade controls and visibility in G Suite

Scale. Speed. Lower total cost of ownership. The cloud offers enterprises significant benefits and gives their teams a freedom that’s as big as their ambitions. G Suite supports such businesses around the world with our unified suite of intelligent apps. Today, more than 3 million paying businesses rely on G Suite to make working together easier, including Whirlpool, PWC, and Woolworths.

Having greater control and visibility when protecting sensitive assets, however, should also be a top concern in today’s world. That’s why starting today, we’re giving customers the critical control and visibility they expect (and their CTOs and regulators often require) in G Suite with:

  • More powerful access control for administrators with security key enforcement
  • More data control with Data Loss Prevention (DLP) for Google Drive, DLP for Gmail, and S/MIME for Gmail
  • More analytics and insights by connecting BigQuery with Gmail

Manage who can access your company’s devices and systems

  • Security key enforcement:  In 2011, we introduced 2-step verification for added protection when users log into their Google accounts. With 2-step verification, admins can opt for their company to use Security Keys, which are both easier for users and more secure against common (phishing) attacks than other methods. Security Keys plug directly into a laptop or pair with a mobile device using Bluetooth or NFC, instead of receiving a code via text. Today, we’re adding the ability for admins to enforce this measure by restricting login solely to users with a Security Key. Admins will also be able to manage the deployment of Security Keys and view usage reports.

See how stripe enhances account security with G Suite

Stripe, a software platform for running an internet business, processes billions of dollars a year for hundreds of thousands of companies around the world. Everyday, Stripe employees rely on G Suite to communicate and collaborate quickly, effectively and securely. Security Key enforcement provides an added layer of protections for Stripe, with advanced authentication controls to combat phishing attempts and unauthorized email access. 

"Businesses that run on Stripe trust us with some of their most sensitive information, and we take the security of that information extremely seriously. We've found Security Keys to be ideal second factors because they balance ease-of-use with increased account security. G Suite Security Key enforcement gives us a way to easily enforce secure logins for all employees, across both their desktop and now mobile devices, " says Bryan Berg, security team member at Stripe.

Control how data is shared beyond your company

  • Data loss prevention (DLP) for Google Drive: In 2015, we launched DLP for Gmail with easy-to-set-up rules, OCR image scan recognition and advanced customization. Today, we’re extending DLP to Google Drive, making it even easier for admins to secure sensitive data, control what content can be stored and protect users from accidentally sharing confidential information externally. G Suite’s DLP protection goes beyond standard DLP with easy-to-configure rules and OCR recognition of content stored in images so admins can easily enforce policies and control how data is shared.

GIf 1

  • S/MIME for Gmail: When it comes to Gmail security, there are dozens of security measures running behind the scenes to keep your emails safe, and we support industry-standard authentication to help combat email spoofing. Unfortunately, other email services that you exchange emails with might not take similar measures to protect your data. That’s why we’re offering additional protection by giving you the option to bring your own certificates for S/MIME encryption. Administrators will be able to enforce S/MIME usage and can set DLP rules based on specific organizational unit needs.

Understand employee behavior and activity

  • Gmail logs + BigQuery integration: Gmail logs contain valuable information that can help administrators diagnose issues or unlock insights. Today, we are making it easier to analyze Gmail logs with a preconfigured BigQuery integration so administrators can run sophisticated, high-performing custom queries, analyze their data and build custom dashboards.

gif 2

  • Third-party email archiving: Google Vault helps you easily archive Gmail content while also complying with retention for legal holds and audit reporting. That said, some organizations use other third-party SMTP archiving solutions like HP Autonomy or Veritas. Today, we’re making it simple for you to integrate third-party archiving solutions of your choice with Gmail.

Part of the new G Suite Enterprise edition, today’s announcement is designed to provide the peace-of-mind our enterprise customers demand and deserve.

See how your business can innovate faster — and now even more securely — with G Suite by Google Cloud.

Source: Drive


SLOs, SLIs, SLAs, oh my – CRE life lessons



Last week on CRE life lessons, we discussed how to come up with a precise numerical target for system availability. We term this target the Service Level Objective (SLO) of our system. Any discussion we have in future about whether the system is running sufficiently reliably and what design or architectural changes we should make to it must be framed in terms of our system continuing to meet this SLO.

We also have a direct measurement of SLO conformance: the frequency of successful probes of our system. This is a Service Level Indicator (SLI). When we evaluate whether our system has been running within SLO for the past week, we look at the SLI to get the service availability percentage. If it goes below the specified SLO, we have a problem and may need to make the system more available in some way, such as running a second instance of the service in a different city and load balancing between the two.

Why have an SLO at all?

Suppose that we decide that running our aforementioned Shakespeare service against a formally defined SLO is too rigid for our tastes; we decide to throw the SLO out of the window and make the service “as available as is reasonable.” This makes things easier, no? You simply don’t mind if the system goes down for an hour now and then. Indeed, perhaps downtime is normal during a new release and the attending stop-and-restart.

Unfortunately for you, customers don’t know that. All they see is that Shakespeare searches that were previously succeeding have suddenly started to return errors. They raise a high-priority ticket with support, who confirms that they see the error rate and escalate to you. Your on-call engineer investigates, confirms this is a known issue, and responds to the customer with “this happens now and again, you don’t have to escalate.” Without an SLO, your team has no principled way of saying what level of downtime is acceptable; there's no way to measure whether or not this a significant issue with the service. and you cannot terminate the escalation early with “Shakespeare search service is currently operating within SLO.” As our colleague Perry Lorier likes to say, “if you have no SLOs, toil is your job.”

The SLO you run at becomes the SLO everyone expects


A common pattern is to start your system off at a low SLO, because that’s easy to meet: you don’t want to run a 24/7 rotation, your initial customers are OK with a few hours of downtime, so you target at least 99% availability  1.68 hours downtime per week. But in fact, your system is fairly resilient and for six months operates at 99.99% availability  down for only a few minutes per month.

But then one week, something breaks in your system and it’s down for a few hours. All hell breaks loose. Customers page your on-call complaining that your system has been returning 500s for hours. These pages go unnoticed, because on-call leaves their pagers on their desks overnight, per your SLO which only specifies support during office hours.

The problem is, customers have become accustomed to your service being always available. They’ve started to build it into their business systems on the assumption that it’s always available. When it’s been continually available for six months, and then goes down for a few hours, something is clearly seriously wrong. Your excessive availability has become a problem because now it’s the expectation. Thus the expression, “An SLO is a target from above and below”  don’t make your system very reliable if you don’t intend and commit to it to being that reliable.

Within Google, we implement periodic downtime in some services to prevent a service from being overly available. In the SRE Book, our colleague Marc Alvidrez tells a story about our internal lock system  Chubby. Then, there’s the set of test front-end servers for internal services to use in testing, allowing those services to be accessible externally. These front-end servers are convenient but are explicitly not intended for use by real services; they have a one business day support SLA, and so can be down for 48 hours before the support team is even obligated to think about fixing them. Over time, experimental services that used those front-ends started to become critical; when we finally had a few hours of downtime on the front-ends, it caused widespread consternation.

Now we run a quarterly planned-downtime exercise with these front-ends. The front-end owners send out a warning, then block all services on the front-ends except for a small whitelist. They keep this up for several hours, or until a major problem with the blockage appears; the blockage can be quickly reversed in that case. At the end of the exercise the front-end owners receive a list of services that use the front-ends inappropriately, and work with the service owners to move them to somewhere more suitable. This downtime exercise keeps the front-end availability suitably low, and detects inappropriate dependencies in time to get them fixed.

Your SLA is not your SLO


At Google, we distinguish between a Service-Level Agreement (SLA) and a Service-Level Objective (SLO). An SLA normally involves a promise to someone using your service that its availability should meet a certain level over a certain period, and if it fails to do so then some kind of penalty will be paid. This might be a partial refund of the service subscription fee paid by customers for that period, or additional subscription time added for free. The concept is that going out of SLA is going to hurt the service team, so they'll push hard to keep it within SLA.

Because of this, and because the principle availability shouldn’t be much better than the SLO, the SLA is normally a looser objective than the SLO. This might be expressed in availability numbers: for instance, an availability SLA of 99.9% over 1 month with an internal availability SLO of 99.95%. Alternatively the SLA might only specify a subset of the metrics comprising the SLO.

For example, with our Shakespeare search service, we might decide to provide it as an API to paying customers in which a customer pays us $10K per month for the right to send up to one million searches per day. Now that money is involved, we need to specify in the contract how available they can expect the service to be, and what happens if we breach that agreement. We might say that we'll provide the service at a minimum of 99% availability, following the definition of successful queries given previously. If the service drops below 99% availability in a month, then we'll refund $2K; if it drops below 80% then, we'll refund $5K.

If you have an SLA that's different from your SLO, as it almost always is, it’s important for your monitoring to measure SLA compliance explicitly. You want to be able to view your system’s availability over the SLA calendar period, and easily see if it appears to be in danger of going out of SLA. You'll also need a precise measurement of compliance, usually from logs analysis. Since we have an extra set of obligations (in the form of our SLA) to paying customers, we need to measure queries received from them separately from other queries (we might not mind dropping queries from non-paying users if we have to start load shedding, but we really care about any query from the paying customer that we fail to handle properly). That’s another benefit of establishing an SLA  it’s an unambiguous way to prioritize traffic.

When you define your SLA, you need to be extra-careful about which queries you count as legitimate. For example, suppose that you give each of three major customers (whose traffic dominates your service) a quota of one million queries per day. One of your customers releases a buggy version of their mobile client, and issues two million queries per day for two days before they revert the change. Over a 30-day period you’ve issued approximately 90 million good responses, and two million errors; that gives you a 97.8% success rate. You probably don’t want to give all your customers a refund as a result of this; two customers had all their queries succeed, and the customer for whom two million out of 32 million queries were rejected brought this upon themselves. So perhaps you should exclude all “out of quota” response codes from your SLA accounting.

On the other hand, suppose you accidentally push an empty quota specification file to your service before going home for the evening. All customers receive a default 1000 queries per day quota. Your three top customers get served constant “out of quota” errors for 12 hours until you notice the problem when you come into work in the morning, and revert the change. You’re now showing 1.5 million rejected queries out of 90 million for the month, a 98.3% success rate. This is all your fault: counting this as 100% success for 88.5M queries is missing the point and a moral failure for measuring the SLA.

Conclusion


SLIs, SLOs and SLAs aren’t just useful abstractions. Without them you cannot know if your system is reliable, available, or even useful. If they don’t tie explicitly back to your business objectives then you have no idea if the choices you make are helping or hurting your business. You also can’t make honest promises to your customers.

If you’re building a system from scratch, make sure that SLIs, SLOs and SLAs are part of your system requirements. If you already have a production system but don’t have them clearly defined then that’s your highest priority work.

To summarize:
  • If you want to have a reliable service, you must first define “reliability.” In most cases that actually translates to availability.
  • If you want to know how reliable your service is, you must be able to measure the rates of successful and unsuccessful queries; these will form the basis of your SLIs.
  • The more reliable the service, the more it costs to operate. Define the lowest level of reliability that you can get away with, and state that as your Service Level Objective (SLO).
  • Without an SLO, your team and your stakeholders cannot make principled judgements about whether your service needs to be made more reliable (increasing cost and slowing development) or less reliable (allowing greater velocity of development).
  • If you’re charging your customers money you'll probably need an SLA, and it should be a little bit looser than your SLO.

As an SRE (or DevOps professional), it's your responsibility to understand how your systems serve the business in meeting those objectives, and, as much as possible, control for risks that threaten the high-level objective. Any measure of system availability that ignores business objectives is worse than worthless because it obfuscates the actual availability, leading to all sorts of dangerous scenarios, false senses of security and failure.

For those of you who wrote us thoughtful comments and questions from our last article, we hope this post has been helpful. Keep the feedback coming!

N. B. Google Cloud Next '17 is fewer than seven weeks away. Register now to join Google Cloud SVP Diane Greene, Google CEO Sundar Pichai, and other luminaries for three days of keynotes, code labs, certification programs, and over 200 technical sessions. And for the first time ever, Next '17 will have a dedicated space for attendees to interact with Google experts in Site Reliability Engineering and Developer Operations.

Testing on the Toilet: Keep Cause and Effect Clear

by Ben Yu

This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.


Can you tell if this test is correct?
208: @Test public void testIncrement_existingKey() {
209: assertEquals(9, tally.get("key1"));
210: }

It’s impossible to know without seeing how the tally object is set up:
1:   private final Tally tally = new Tally();
2: @Before public void setUp() {
3: tally.increment("key1", 8);
4: tally.increment("key2", 100);
5: tally.increment("key1", 0);
6: tally.increment("key1", 1);
7: }
// 200 lines away
208: @Test public void testIncrement_existingKey() {
209: assertEquals(9, tally.get("key1"));
210: }

The problem is that the modification of key1's values occurs 200+ lines away from the assertion. Otherwise put, the cause is hidden far away from the effect.

Instead, write tests where the effects immediately follow the causes. It's how we speak in natural language: “If you drive over the speed limit (cause), you’ll get a traffic ticket (effect).” Once we group the two chunks of code, we easily see what’s going on:
1:   private final Tally tally = new Tally();
2: @Test public void testIncrement_newKey() {
3: tally.increment("key", 100);
5: assertEquals(100, tally.get("key"));
6: }
7: @Test public void testIncrement_existingKey() {
8: tally.increment("key", 8);
9: tally.increment("key", 1);
10: assertEquals(9, tally.get("key"));
11: }
12: @Test public void testIncrement_incrementByZeroDoesNothing() {
13: tally.increment("key", 8);
14: tally.increment("key", 0);
15: assertEquals(8, tally.get("key"));
16: }

This style may require a bit more code. Each test sets its own input and verifies its own expected output. The payback is in more readable code and lower maintenance costs.