Tag Archives: Security & Identity

Improving our account management policies to better support customers

Recently, a Google Cloud Platform (GCP) customer blogged about an incident in June, in which a project they were running on Google Cloud Platform was suspended. We really appreciated the candid feedback, in which our customer noted several aspects of our account management process which needed to be improved. We also appreciated our customer’s followup and recognition of the Google Cloud support team, “who have reached out and assured us these incidents will not repeat.”

Here’s what we are doing to be as good as our word, and provide a more careful, accurate, thoughtful and empathetic account management experience for our GCP customers. These changes are intended to provide peace of mind and a predictable, positive experience for our GCP customers, while continuing to permit appropriate suspension and removal actions for the inevitable bad actors and fraud which are a part of operating a public cloud service.

No Automatic Fraud-Based Account Suspensions for Established Customers with Offline Payment. Established GCP customers complying with our Acceptable Use Policy (AUP, TOS and local laws), with an offline billing contract, invoice billing, or an active relationship with our sales team, are not subject to fraud-based automatic account suspension.

Delayed Suspension for Established Online Customers. Online customers with established payment history, operating in compliance with our TOS, AUP and local laws, will receive advance notification and a 5 day cure period in the event that fraud or account compromise activity is detected in their projects.

Other Customers. For all other customers, we will institute a second human review for flagged fraud accounts prior to suspending an account. We’re also modifying who has authority to suspend an account, as well as refreshing our training for the teams that review flagged accounts and determine response actions; re-evaluating the signals, sources, and the tools we use to assess potential fraudulent activity; and increasing the number of options we can use to help new customers quickly and safely grow their usage while building an account history with Google.

In addition to the above, for all customers we are making the following improvements:

24X7 Chat Support. We are rolling out 24X7 chat support for customers that receive account notices, so that customers can always reach us easily. We expect this to be fully rolled out for all customers by September.

Correcting Notices About our 30 Day Policy. Our customer noted, with appropriate concern, that their suspension email stated “we will delete your account in 3 days.” This language was simply incorrect -- our fraud suspension policy provides 30 days before removal. We have corrected the communication language, and we are conducting a full review of our communication verbiage and systems and ensuring that our messages are accurate and clear.

Updating Our Project Suspension Guidelines. We will review and update our project suspension guidelines to clarify our practices and describe what you should expect from Google.

Improving Customer Contact Points. We will encourage customers to provide us with a verifiable phone number, email, and other contact channels, both at sign-up and at later points in time, so that we can quickly contact you if we detect suspicious activity on your account.

Creating Customer Pre-Verification. We will provide ways for customers to pre-verify their accounts with us if they desire, either at sign-up or at a later point in time.

These suspensions are our responsibility.There are also steps that customers can take to help us protect their accounts including:
  1. Make sure to monitor emails sent to your payments and billing contacts so you don’t miss important alerts.
  2. Provide a valid phone number where we can reach you in the event of suspicious activity on your account.
  3. Provide one or more billing admins to your account.
  4. Provide a secondary payment method in case there are problems charging your primary method.
  5. Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards.
We’re making immediate changes to ensure our policies will improve our customer’s experience. Our work here is never done and we will continue to update and optimize based on your feedback.

We sincerely apologize to all our customers who’ve been concerned or had to go through a service reinstatement. Please keep the feedback coming, we’ll work to continue to earn your trust every day.

Introducing Endpoint Verification: visibility into the desktops accessing your enterprise applications

While corporate devices are the key to employee productivity, they can also be the weak link when it comes to application and data security. Today we are introducing Endpoint Verification, giving admins an overview of the security posture of their laptop and desktop devices. Having that inventory of what computers employees are using provides valuable information which the enterprise can use to maintain security. Available to all Google Cloud Platform (GCP), Cloud Identity, G Suite Business, and G Suite Enterprise customers, Endpoint Verification consists of a Chrome extension and native app and is available for ChromeOS, macOS, and Windows devices.
Endpoint Verification is available as a Chrome extension

With the proliferation of multiple platforms and bring your own device (BYOD) in the enterprise, administrators find full MDM solutions to be difficult to deploy and maintain. Endpoint Verification offers a lightweight, easy-to-deploy solution to desktop device reporting for GCP, Cloud Identity and G Suite customers.

With Endpoint Verification, enterprises get two key value adds immediately. First, you can now build an inventory of devices within the enterprise that access corporate data. And second, with Endpoint Verification, admins have access to device information including: screen lock, disk encryption, and OS version.

For information on how to deploy Endpoint Verification, please visit the help center. For organizations that would like to try this out, a free trial of Cloud Identity is available here.

Protect your Compute Engine resources with keys managed in Cloud Key Management Service

In Google Cloud Platform, customer data stored at rest is always encrypted by default using multiple layers of encryption technology. We also offer a continuum of encryption key management options to help meet the security requirements of your organization.

Did you know there is now beta functionality you can use to further increase protection of your Compute Engine disks, images and snapshots using your own encryption keys stored and managed with Cloud Key Management Service (KMS)? These customer-managed encryption keys (CMEKs) provide you with granular control over which disks, images and snapshots will be encrypted.

You can see below that on one end of the spectrum, Compute Engine automatically encrypts disks and manages keys on your behalf. On the other end, you can continue using your customer-supplied encryption keys (CSEK) for your most sensitive or regulated workloads, if you desire.

This feature helps you strike a balance between ease of use and control: Your keys are in the cloud, but still under your management. This option is ideal for organizations in regulated industries that need to meet strict requirements for data protection, but which don’t want to deal with the overhead of creating custom solutions to generate and store keys, manage and record key access, etc.

Setting up CMEK in Compute Engine helps quickly deliver peace of mind to these organizations, because they control access to the disk by virtue of controlling the key.

How to create a CMEK-supported disk

Getting started with the CMEK feature is easy. Follow the steps below to create and attach a Compute Engine disk that is encrypted with a key that you control.

You’ll need to create a key ring and key in KMS. This—and all the rest of the steps below—can be accomplished in several ways: through the Developer Console, APIs and gcloud. In this tutorial, we’ll be using the developer console. We’ll start on the Cryptographic Keys page, where we’ll select “Create Key Ring.”

Give your keyring a name. Do the same with the key the next page. For this tutorial, feel free to leave all the other fields as-is.

Having finished those steps, you now have a keyring with a single AES-256 encryption key. In the screenshot below, you can see it as “tutorial-keyring-1.” And since the keyring is managed by KMS, it is already fully integrated with Cloud Identity and Access Management (IAM) and Cloud Audit Logging, so you can easily manage permissions and monitor how it’s used.

With the key in place, you can start encrypting disks with CMEK keys. The instructions below are for creating a new instance and protecting its boot disk with a CMEK key. Note that it is also possible to create new encrypted disks from snapshots and images and attach them to existing VMs, or even to encrypt the underlying snapshots and images themselves.

First we’ll go to the VM instances page, and create a new VM instance.

On the instance creation page, expand the “Management, Disks, Networking and SSH keys” section and go to the “Disks” tab. There, you’ll see options for the three different encryption options described above. Select “Custom-managed key” and select the appropriate key from the dropdown menu.

Note that if this is your first time doing this, you may see the following dialogue. This is expected - you’ll need to grant this service account permissions. In turn, this service account is used by Compute Engine to do the actual encryption and decryption of disks, images and snapshots.

Once you’ve done this, confirm the VM creation by selecting “Create”.

And there you have it! With a few easy steps, you can create a key in Cloud KMS, encrypt a disk with the key and mount it to a VM. Since you manage the key, you can choose at any time to suspend or delete the key. If that happens, resources protected by that key won’t start until key access is restored.

Try CMEKs for yourself

Visit the Developer Console and start using your CMEKs to help secure your Compute Engine disks, images and snapshots. As always, your feedback is invaluable to us, and we’d love to hear what you think. Safe computing!

Six essential security sessions at Google Cloud Next 18

We aim to be the most secure cloud, but what does that mean? If you’re coming to Google Cloud Next '18 next month in San Francisco, now is your chance to identify and understand the technologies and best practices that set Google Cloud Platform (GCP) apart from other cloud providers. There are dozens of breakout sessions dedicated to security, but if time is short, here are six sessions that will give you a solid understanding of foundational GCP security practices and offerings, as well as insight into the cutting-edge security research and development being done by our team.

1. How Google Protects Your Data at Rest and in Transit

First things, first. Come learn how Google protects your data within Google infrastructure, when it’s stored on disk as well as when it moves across our network, for use by various services. Google Cloud Security and Privacy Product Managers Maya Kaczorowski and Il-Sung Lee will also cover additional protections you can put in place such as Customer-Managed Encryption Keys, IPsec tunnels, and Istio. More details are available here.

2. How Google's Security Infrastructure Design Enabled Rapid, Seamless Response to “Spectre” and “Meltdown”

Not content to sit back and wait, Google has a huge team of security researchers that actively push the limits of our systems. This year, researchers found two significant vulnerabilities in modern compute architectures: Spectre and Meltdown. This session will detail those vulnerabilities, and more to the point, how we remediated them transparently, without customer downtime. Learn more here.

3. BeyondCorp Beyond Google

New Google employees always marvel at how they can access Google resources from anywhere, without a VPN. That’s made possible by our BeyondCorp model, and core BeyondCorp technologies such as global scale security proxies, phishing-resistant 2nd factor authentication, and laptop security enforcement are increasingly available to Google Cloud customers. In this session, French resource management provider VEOLIA describes how it built out a BeyondCorp model on Google Cloud to reach 169,000 employees across five continents. Register for the session here.

4. Trust Through (Access) Transparency

'When do you access my data, and how will I know?' is a question that troubles every cloud customer who cares about their data—and one that few cloud providers have an answer for. This talk reviews Google's robust data protection infrastructure, and introduces Google's new Access Transparency product, which gives customers near-real-time oversight over data accesses by Google's administrators. The talk also guides customers through how to audit accesses and mitigate against this risk, together with examples from our customers of where this has successfully been done. Register for the session here.

5. Google Cloud: Data Protection and Regulatory Compliance

Security in the cloud is much more than encryption and firewalls. If you’re subject to regulations, you often need to demonstrate data protection and compliance with a variety of regulatory standards. In this session, we cover recent trends in the data protection space, such as GDPR, and share tools you can leverage to help address your compliance needs. You'll learn how you can partner with Google to enhance data security and meet global regulatory obligations. You can find a full session description here.

6. Shield Your Cloud with Verifiable Advanced Platform Security

Last but not least, you’ll definitely want to attend this session by Googlers Andrew Honig and
Nelly Porter, as they discuss issues facing VM security in the cloud, and an interesting new approach to mitigate against local code gaining escalation privileges. After attending this session, you’ll understand how we prevent workloads running on Google Cloud Platform from being penetrated by boot malware or firmware rootkits. Register for the session here.

Of course, this is just the tip of the iceberg. Security runs through everything we do at Google Cloud. In addition to these six sessions, there are 31 other breakout sessions dedicated to security, not to mention keynotes and supersessions, hands-on labs, meetups and bootcamps. Don’t delay, register for Next today.

GCP arrives in the Nordics with a new region in Finland

Click here for the Finnish version, thank you!

Our sixteenth Google Cloud Platform (GCP) region, located in Finland, is now open for you to build applications and store your data.

The new Finland region, europe-north1, joins the Netherlands, Belgium, London, and Frankfurt in Europe and makes it easier to build highly available, performant applications using resources across those geographies.

Hosting applications in the new region can improve latencies by up to 65% for end-users in the Nordics and by up to 88% for end-users in Eastern Europe, compared to hosting them in the previously closest region. You can visit www.gcping.com to see for yourself how fast the Finland region is from your location.


The Nordic region has everything you need to build the next great application, and three zones that allow you to distribute applications and storage across multiple zones to protect against service disruptions.

You can also access our Multi-Regional services in Europe (such as BigQuery) and all the other GCP services via the Google Network, the largest cloud network as measured by number of points of presence. Please visit our Service Specific Terms to get detailed information on our data storage capabilities.

Build sustainably

The new region is located in our existing data center in Hamina. This facility is one of the most advanced and efficient data centers in the Google fleet. Our high-tech cooling system, which uses sea water from the Gulf of Finland, reduces energy use and is the first of its kind anywhere in the world. This means that when you use this region to run your compute workloads, store your data, and develop your applications, you are doing so sustainably.

Hear from our customers

“The road to emission-free and sustainable shipping is a long and challenging one, but thanks to exciting innovation and strong partnerships, Rolls-Royce is well-prepared for the journey. For us being able to train machine learning models to deliver autonomous vessels in the most effective manner is key to success. We see the Google Cloud for Finland launch as a great advantage to speed up our delivery of the project.”
– Karno Tenovuo, Senior Vice President Ship Intelligence, Rolls-Royce

“Being the world's largest producer of renewable diesel refined from waste and residues, as well as being a technologically advanced refiner of high-quality oil products, requires us to take advantage of leading-edge technological possibilities. We have worked together with Google Cloud to accelerate our journey into the digital future. We share the same vision to leave a healthier planet for our children. Running services on an efficient and sustainably operated cloud is important for us. And even better that it is now also available physically in Finland.”
– Tommi Touvila, Chief Information Officer, Neste

“We believe that technology can enhance and improve the lives of billions of people around the world. To do this, we have joined forces with visionary industry leaders such as Google Cloud to provide a platform for our future innovation and growth. We’re seeing tremendous growth in the market for our operations, and it’s essential to select the right platform. The Google Cloud Platform cloud region in Finland stands for innovation.”
– Anssi Rönnemaa, Chief Finance and Commercial Officer, HMD Global

“Digital services are key growth drivers for our renewal of a 108-year old healthcare company. 27% of our revenue is driven by digital channels, where modern technology is essential. We are moving to a container-based architecture running on GCP at Hamina. Google has a unique position to provide services within Finland. We also highly appreciate the security and environmental values of Google’s cloud operations.”
– Kalle Alppi, Chief Information Officer, Mehiläinen

Partners in the Nordics

Our partners in the Nordics are available to help design and support your deployment, migration and maintenance needs.

"Public cloud services like those provided by Google Cloud help businesses of all sizes be more agile in meeting the changing needs of the digital era—from deploying the latest innovations in machine learning to cost savings in their infrastructure. Google Cloud Platform's new Finland region enables this business optimization and acceleration with the help of cloud-native partners like Nordcloud and we believe Nordic companies will appreciate the opportunity to deploy the value to their best benefit.”
– Jan Kritz, Chief Executive Officer, Nordcloud

Nordic partners include: Accenture, Adapty, AppsPeople, Atea, Avalan Solutions, Berge, Cap10, Cloud2, Cloudpoint, Computas, Crayon, DataCenterFinland, DNA, Devoteam, Doberman, Deloitte, Enfo, Evry, Gapps, Greenbird, Human IT Cloud, IIH Nordic, KnowIT, Koivu Solutions, Lamia, Netlight, Nordcloud, Online Partners, Outfox Intelligence AB, Pilvia, Precis Digital, PwC, Quality of Service IT-Support, Qvik, Skye, Softhouse, Solita, Symfoni Next, Soprasteria, Tieto, Unifoss, Vincit, Wizkids, and Webstep.

If you want to learn more or wish to become a partner, visit our partners page.

Getting started

For additional details on the region, please visit our Finland region page where you’ll get access to free resources, whitepapers, the "Cloud On-Air" on-demand video series and more. Our locations page provides updates on the availability of additional services and regions. Contact us to request access to new regions and help us prioritize what we build next.

Introducing sole-tenant nodes for Google Compute Engine — when sharing isn’t an option

Today we are excited to announce beta availability of sole-tenant nodes on Google Compute Engine. Sole-tenant nodes are physical Compute Engine servers designed for your dedicated use. Normally, VM instances run on physical hosts that may be shared by many customers.  With sole-tenant nodes, you have the host all to yourself.

You can launch instances using the same options you would use for regular compute instances, except on server capacity dedicated to you. You can launch instances of any shape (i.e., vCPU and memory). A placement algorithm automatically finds the optimal location to launch your instance across all your nodes. If you prefer more control, you can manually select the location upon which to launch your instances. Instances launched on sole-tenant nodes can take advantage of live migration to avoid downtime during proactive maintenance. Pricing remains simple--pay only for the nodes you use on a per-second basis with a one-minute minimum charge. Sustained use discounts automatically apply, as do any new or existing committed use discounts.

Sole-tenant nodes enable a number of valuable use cases:

  • Compliance and regulation - Organizations with strict compliance and regulatory requirements can use sole-tenant nodes with VM placement to facilitate physical separation of their compute resources in the cloud. 
  • Isolation and utilization - Control instance placement directly via user-defined labels, or let Compute Engine automatically handle instance placement across nodes. You can also create and launch different machine types, or “shapes” on your nodes, in order to achieve the highest level of utilization. 

It’s easy to get started with sole-tenant nodes. You can launch a VM onto a sole-tenant node from the Google Cloud SDK, as well as from the Compute Engine APIs (support for Google Cloud Console coming soon):

gcloud beta compute sole-tenancy node-templates create mynodetemplate
--node-type n1-node-96-624 --region us-central1 

gcloud beta compute sole-tenancy node-groups create mynodegroup
--node-template mynodetemplate --target-size 2 --zone us-central1-a

gcloud beta compute instances create my-vm --node-group
mynodegroup --custom-cpu 4 --custom-memory 8 --zone us-central1-a
For guidance on manual placements and more, check out the documentation. Visit the pricing page to learn more about the offering, as well as regional availability.

7 tips to maintain security controls in your GCP DR environment

Cloud computing has changed many traditional IT practices, and one particularly useful change has been in the area of disaster recovery (DR). Our team helps Google Cloud Platform (GCP) users build their infrastructures with cloud, and we’ve seen some great results when they use GCP as the DR target environment.

When you integrate a cloud provider like GCP into your DR plan, you no longer have to invest up front in mostly idle backup infrastructure. Testing that DR plan no longer seems so daunting, as you can bring up your DR environment automatically and close it all down again when it’s no longer needed—and it’s always ready for the next tests. In the event of an actual disaster, the DR environment can be made ready.

However, planning and testing for and recovering from a disaster involves more than just getting your application restored and available within your Recovery Time Objective (RTO). You need to ensure the security controls you implemented on-premises also apply to your recovered environment. This post provides tips to help you maintain your security controls in your cloud DR environment.

1. Grant users the same access they’re used to

If your production environment is running on GCP, it’s easy to replicate the identity and access management (IAM) policies already in place. Use infrastructure as code methods and employ tools such as Cloud Deployment Manager to recreate your IAM policies. You can then bind the policies to corresponding resources when you’re standing up the DR environment.

If your production environment is on-premises, map functional roles such as your network administrator and auditor roles to appropriate IAM policies. Use the IAM documentation, which has some example configurations such as the networking and audit logging functional roles. You’ll want to configure IAM policies to grant appropriate permissions to GCP products. For example, you might want to restrict access to specific Google Cloud Storage buckets.

Once you’ve created a test environment, ensure that the access granted to your users confers the same permissions as those they are granted on-premises.

If your production environment runs on another cloud, you will need to understand how to map the IAM controls to Google Cloud Identity and Access Management (IAM) policies. The GCP for AWS professionals management doc can help if your other cloud is AWS.

2. Ensure user understanding

Do not wait for a disaster to occur before checking that your users—whether developers, operators, data scientists, security or network admins—can access the DR environment. Make sure their accounts have been granted the appropriate access rights. If you are using an alternative identity system, ensure accounts have been synced with your Cloud Identity account. Make sure users who will need access to the DR environment (which will be the production environment for awhile) are able to log in to the DR environment. Resolve any authentication issues. When you’re conducting regular DR tests, incorporate users logging into the DR environment as part of the test process.

Enable the GCP OS login feature on the projects that constitute your DR environment so you can centrally manage who has SSH access to VMs launched in the DR environment.

It’s also important to train users on the DR environment so that they understand how to undertake their usual actions in GCP. Use the test environment for this.

3. Double-check blocking and compliance requirements

It’s essential that your network controls confer the same separation and blocking settings in the DR as the source production environment. Learn how to configure Shared VPCs and GCP firewalls and take advantage of using service accounts as part of the firewall rules. Understand how to use service accounts to implement least privileges for applications accessing GCP APIs.

In addition, make sure your DR environment meets your compliance requirements. Validate that access to your DR environment is restricted to only those who need access. Ensure personally identifiable information (PII) data is appropriately redacted and encrypted. If you carry out regular penetration tests on your production environment, you should start including your DR environment and carry out those tests by regularly standing up a DR environment.

4. Capture log data

When the DR environment is in service, the logs collected during that time should be backfilled into your production environment log archive. Ensure that as part of your GCP DR environment you are able to export audit logs that are collected via Stackdriver to your main log sink archive. Use the export sink facilities. For application logs, stand up a mirror of your on-premises logging and monitoring environment. For another cloud, map across to the equivalent GCP services. Have a process in place to format this log input into your production environment.

5. Use Cloud Storage for day-to-day backup routines

Use Cloud Storage to store backups that result from the DR environment. Ensure the storage buckets containing your backups have limited permissions applied to them.

The same security controls should apply to your recovered data as if it were production data. The same permissions, encryption and audit requirements apply. Know where your backups are located and where, and who restored them.

6. Consider secrets and key management

Manage application-level secrets and keys using GCP to host the key or secret management service. You can use Cloud Key Management Service (KMS) or a third-party solution like HashiCorp Vault with a GCP backend such as Cloud Spanner or Cloud Storage.

7. Manage VM images and snapshots

If you have predefined configurations for VMs, such as who can use them or make changes, reflect those controls in your GCP DR recovery site. Follow the guidance outlined in restricting access to images.

These tips can help make sure you don’t lose the security you’ve built into your production environment when you stand up a DR site. You’ll be on your way more quickly to having a cloud DR site that works for your users and the business.

Next steps:

Read our guide on How to Design a DR plan.

Related content:

Cloud Identity-Aware Proxy: a simple and more secure way to manage application access
Know thy enemy: how to prioritize and communicate risks - CRE life lessons
Building trust through Access Transparency

Google Cloud named a leader in latest Forrester Research Public Cloud Platform Native Security Wave

Today, we are happy to announce that Forrester Research has named Google Cloud as one of just two leaders in The Forrester WaveTM: Public Cloud Platform Native Security (PCPNS), Q2 2018 report, and rated Google Cloud highest in strategy. The report evaluates the native security capabilities and features of public cloud providers, such as encryption, identity and access management (IAM) and workload security.

The report finds that most security and risk professionals (S&R) now believe that “native security capabilities of large public cloud platforms actually offer more affordable and superior security than what S&R teams could deliver themselves if the workloads remained on-premises.”

The report particularly highlights that “Google has been continuing to invest in PCPNS. The platform’s security configuration policies are very granular in the admin console as well as in APIs. The platform has a large number of security certifications, broad partner ecosystem, offers deep native support for guest operating systems and Kubernetes containers, and supports auto-scaling (GPUs can be added to instances).”

In this wave, Forrester evaluated seven public cloud platforms against 37 criteria, looking at current offerings, strategy and market presence. Of the seven vendors, Google Cloud scored highest on strategy, and received the highest score possible in its strategic plans in physical security, certifications and attestations, hypervisor security, guest OS workload security, network security, and machine learning criteria.

Further, Forrester cited Google Cloud’s security roadmap. As part of our roadmap, Google Cloud continues to redefine what’s possible in the cloud with unique security capabilities like Access Transparency, Istio, Identity-Aware Proxy, VPC Service Controls, and Asylo.

“The vendor plans: to 1) provide ongoing security improvements to the admin console using device trust, location, etc., 2) implement hardware-backed encryption key management, and 3) improve visibility into the platform by launching a unified risk dashboard."

At Google, we have worked for over a decade to build a secure, scalable and flexible cloud foundation. Our belief is that if you put security first, everything else will follow. Security continues to be top of mind—from our custom hardware like our Titan chip, to data encryption both at rest and in transit by default. On this strong foundation, we offer enterprises a rich set of controls and capabilities to meet their security and compliance requirements.

You can download the full Forrester Public Cloud Platform Native Security Wave Q2 2018 report here. To learn more about GCP, visit our website, and sign up for a free trial.

Introducing Shared VPC for Google Kubernetes Engine

Containers have come a long way, from the darling of cloud-native startups to the de-facto way to deploy production workloads in large enterprises. But as containers grow into this new role, networking continues to pose challenges in terms of management, deployment and scaling.

We are excited to introduce Shared Virtual Private Cloud (VPC) to help Google Kubernetes Engine tackle scale and reliability needs of container networking for enterprise workloads.

Shared VPC for better control of enterprise network resources
In large enterprises, you often need to place different departments into different projects, for purposes of budgeting, access control, accounting, etc. And while isolation and network segmentation are recommended best practices, network segmentation can pose a challenge for sharing resources. In Compute Engine environments, Shared VPC networks let enterprise administrators give multiple projects permission to communicate via a single, shared virtual network without giving control of critical resources such as firewalls. Now, with the general availability of Kubernetes Engine 1.10, we are extending the shared VPC model to connect Kubernetes Engine clusters with versions 1.8 and above—connecting from multiple projects to a common VPC network.

Before Shared VPC, it was possible to achieve this setup in a crippled, insecure way, by bridging projects in their own VPC with Cloud VPNs. The problem with this approach was that it required N*(N-1)/2 connections to obtain full connectivity between each project. An additional challenge was that the network for each cluster wasn’t fully configurable, making it difficult for one project to communicate with another without a NAT gateway in between. Security was another concern since the organization administrator had no control over firewalls in the other projects.

Now, with Shared VPC, you can overcome these challenges and compartmentalize Kubernetes Engine clusters into separate projects, for the following benefits to your enterprise:
  • Sharing of common resources - Shared VPC makes it easy to use network resources that must be shared across various teams, such as a set of IPs within RFC 1918 shared across multiple subnets and Kubernetes Engine clusters.
  • Security - Organizations want to leverage more granular IAM roles to separate access to sensitive projects and data. By restricting what individual users can do with network resources, network and security administrators can better protect enterprise assets from inadvertent or deliberate acts that can compromise the network. While a network administrator can set firewalls for every team, a cluster administrator’s responsibilities might be to manage workloads within a project. Shared VPC provides you with centralized control of critical network resources under the organization administrator, while still giving flexibility to the various organization admins to manage clusters in their own projects.
  • Billing - Teams can use projects and separate Kubernetes Engine clusters to isolate their resource usage, which helps with accounting and budgeting needs by letting you view billing for each team separately.
  • Isolation and support for multi-tenant workloads - You can break up your deployment into projects and assign them to the teams working on them, lowering the chances that one team’s actions will inadvertently affect another team’s projects.

Below is an example of how you can map your organizational structure to your network resources using Shared VPC:

Here is what, Spotify, one of the many enterprises using Kubernetes Engine, has to say:
“Kubernetes is our preferred orchestration solution for thousands of our backend services because of its capabilities for improved resiliency, features such as autoscaling, and the vibrant open-source community. Shared VPC in Kubernetes Engine is essential for us to be able to use Kubernetes Engine with our many GCP projects."
- Matt Brown, Software Engineer, Spotify

How does Shared VPC work?
Host project contains one or more shared network resources while the service project(s) map to the different teams or departments in your organization. After setting up the correct IAM permissions for service accounts in both the host and service projects, the cluster admin can instantiate a number of Compute Engine resources in any of the service projects. This way, critical resources like firewalls are centrally managed by the network or security admin, while cluster admins are able to create clusters in the respective service projects.

Shared VPC is built on top of Alias IP. Kubernetes Engine clusters in service projects will need to be configured with a primary CIDR range (from which to draw Node IP addresses), and two secondary CIDR ranges (from which to draw Kubernetes Pod and Service IP addresses). The following diagram illustrates a subnet with the three CIDR ranges from which the clusters in the Shared VPC are carved out.

The Shared VPC IAM permissions model
To get started with Shared VPC, the first step is to set up the right IAM permissions on service accounts. For the cluster admin to be able to create Kubernetes Engine clusters in the service projects, the host project administrator needs to grant the compute.networkUser and container.hostServiceAgentUser roles in the host project, allowing the service project's service accounts to use specific subnetworks and to perform networking administrative actions to manage Kubernetes Engine clusters. For more detailed information on the IAM permissions model, follow along here.

Try it out today!
Create a Shared VPC cluster in Kubernetes Engine and get the ease of access and scale for your enterprise workloads. Don’t forget to sign up for the upcoming webinar, 3 reasons why you should run your enterprise workloads on Kubernetes Engine.

Exploring container security: Isolation at different layers of the Kubernetes stack

Editor’s note: This is the seventh in a series of blog posts on container security at Google.

To conclude our blog series on container security, today’s post covers isolation, and when containers are appropriate for actually, well... containing. While containers bring great benefits to your development pipeline and provide some resource separation, they were not designed to provide a strong security boundary.

The fact is, there are some cases where you might want to run untrusted code in your environment. Untrusted code lives on a spectrum. On one end you have known bad malware that an engineer is trying to examine and reverse-engineer; and on the other end you might just have a third-party application or tool that you haven't audited yourself. Maybe it’s a project that historically had vulnerabilities and you aren’t quite ready to let it loose in your environment yet. In each of these cases, you don’t want the code to affect the security of your own workloads.

With that said, let’s take a look at what kind of security isolation containers do provide, and, in the event that it’s not enough, where to look for stronger isolation.

Hypervisors provide a security boundary for VMs
Traditionally, you might have put this untrusted code in its own virtual machine, relying on the security of the hypervisor to prevent processes from escaping or affecting other processes. A hypervisor provides a relatively strong security boundary—that is, we don’t expect code to be able to easily cross it by breaking out of a VM. At Google, we use the KVM hypervisor, and put significant effort into ensuring its security.

The level of trust you require for your code is all relative. The more sensitive the data you process, the more you need to be able to trust the software that accesses it. You don’t treat code that doesn’t access user data (or other critical data) the same way you treat code that does—or that’s in the serving path of active user sessions, or that has root cluster access. In a perfect world, you access your most critical data with code you wrote, reviewed for security issues, and ran some security checks against (such as fuzzing). You then verify that all these checks passed before you deploy it. Of course, you may loosen these requirements based on where the code runs, or what it does—the same open-source tool might be insufficiently trusted in a hospital system to examine critical patient information, but sufficiently trusted in a test environment for a game app you’re developing in your spare time.

A ‘trust boundary’ is the point at which your code changes its level of trust (and hence its security requirements), and a ‘security boundary’ is how you enforce these trust boundaries. A security boundary is a set of controls, managed together across all surfaces, to prevent a process from one trust level from elevating its trust level and affecting more trusted processes or access other users’ data. A container is one such security boundary, albeit not a very strong one. This is because, compared to a hypervisor, a native OS container is a larger, more complex security boundary, with more potential vulnerabilities. On the other hand, containers are meant to be run on a minimal OS, which limits the potential surface area for an attack at the OS level. At Google, we aim to protect all trust boundaries with at least two different security boundaries that each need to fail in order to cross a trust boundary.

Layers of isolation in Kubernetes
Kubernetes has several nested layers, each of which provides some level of isolation and security. Building on the container, Kubernetes layers provide progressively stronger isolation— you can start small and upgrade as needed. Starting from the smallest unit and moving outwards, here are the layers of a Kubernetes environment:

  • Container (not specific to Kubernetes): A container provides basic management of resources, but does not isolate identity or the network, and can suffer from a noisy neighbor on the node for resources that are not isolated by cgroups. It provides some security isolation, but only provides a single layer, compared to our desired double layer.
  • Pod: A pod is a collection of containers. A pod isolates a few more resources than a container, including the network. It does so with micro-segmentation using Kubernetes Network Policy, which dictates which pods can speak to one another. At the moment, a pod does not have a unique identity, but the Kubernetes community has made proposals to provide this. A pod still suffers from noisy neighbors on the same host.
  • Node: This is a machine, either physical or virtual. A node includes a collection of pods, and has a superset of the privileges of those pods. A node leverages a hypervisor or hardware for isolation, including for its resources. Modern Kubernetes nodes run with distinct identities, and are authorized only to access the resources required by pods that are scheduled to the node. There can still be attacks at this level, such as convincing the scheduler to assign sensitive workloads to the node. You can use firewall rules to restrict network traffic to the node.
  • Cluster: A cluster is a collection of nodes and a control plane. This is a management layer for your containers. Clusters offer stronger network isolation with per-cluster DNS.
  • Project: A GCP project is a collection of resources, including Kubernetes Engine clusters. A project provides all of the above, plus some additional controls that are GCP-specific, like project-level IAM for Kubernetes Engine and org policies. Resource names, and other resource metadata, are visible up to this layer.
There’s also the Kubernetes Namespace, the fundamental unit for authorization in Kubernetes. A namespace can contain multiple pods. Namespaces provide some control in terms of authorization, via namespace-level RBAC, but don’t try to control resource quota, network, or policies. Namespaces allow you to easily allocate resources to certain processes; these are meant to help you manage how you use your resources, not necessarily prevent a malicious process from escaping and accessing another process’ resources.

Diagram 1: Isolation provided by layer of Kubernetes

Recently, Google also announced the open-source gVisor project, which provides stronger isolation at the pod level.

Sample scenario: Multi-tenant SaaS workload
In practice, it can be hard to decide what isolation requirements you should have for your workload, and how to enforce them—there isn’t a one-size-fits-all solution. Time to do a little threat modeling.

A common scenario we hear, is a developer building a multi-tenant SaaS application running in Kubernetes Engine, in order to help manage and scale their application as needed to meet demand. In this scenario, let’s say we have a SaaS application running its front-end and back-end on Kubernetes Engine, with a back-end database for transaction data, and a back-end database for payment data; plus some open-source code for critical functions such as DNS and secret management.

You might be worried about a noisy (or nosy!) neighbor—that someone else is monopolizing resources you need, and you’re unable to serve your app. Cryptomining is a trendy attack vector these days, and being able to stay up and running even if one part of your infrastructure is affected is important to you. In cases like these, you might want to isolate certain critical workloads at the node layer.

You might be worried about information leaking between your applications. Of course Spacely Sprockets knows that you have other customers, but it shouldn’t be able to find out that Cogswell’s Cogs is also using your application—they’re competitors. In this case, you might want to be careful with your naming, and take care to block access to unauthenticated node ports (with NetworkPolicy), or isolate at the cluster level.

You might also be concerned that critical data, like customer payment data, is sufficiently segmented from access by less trusted workloads. Customer payment data should require different trust levels to access than user-submitted jobs. In this case, you might want to isolate at the cluster level, or run these each in their own sandbox.

So all together, you might have your entire application running in a single project, with different clusters for each environment, and place any highly trusted workload in its own cluster. In addition, you’ll need to make careful resource sharing decisions at the node and pod layers to isolate different customers.

Another common multi-tenant scenario we hear is one where you’re running entirely untrusted code. For example, your users may give you arbitrary code that you run on their behalf. In this case, for a multi-tenant cluster you'll want to investigate sandboxing solutions.

In the end
If you’ve learned one thing from this blog post, it’s that there’s no one right way to configure a Kubernetes environment—the right security isolation settings depend on what you are running, where, who is accessing the data, and how. We hope you enjoyed this series on container security! And while this is the last installment, you can look forward to more information about security best practices, as we continue to make Google Cloud, including Kubernetes Engine, the best place to run containers.