Tag Archives: Security & Identity

Introducing Asylo: an open-source framework for confidential computing



Protecting data is the number one consideration when running workloads in the cloud. While cloud infrastructures offer numerous security controls, some enterprises want additional verifiable isolation for their most sensitive workloads—capabilities which have become known as confidential computing. Today we’re excited to announce Asylo (Greek for “safe place”), a new open-source framework that makes it easier to protect the confidentiality and integrity of applications and data in a confidential computing environment.

Asylo is an open source framework for confidential computing


Asylo is an open-source framework and SDK for developing applications that run in trusted execution environments (TEEs). TEEs help defend against attacks targeting underlying layers of the stack, including the operating system, hypervisor, drivers, and firmware, by providing specialized execution environments known as “enclaves”. TEEs can also help mitigate the risk of being compromised by a malicious insider or an unauthorized third-party. Asylo includes features and services for encrypting sensitive communications and verifying the integrity of code running in enclaves, which help protect data and applications.

Previously, developing and running applications in a TEE required specialized knowledge and tools. In addition, implementations have been tied to specific hardware environments. Asylo makes TEEs much more broadly accessible to the developer community, across a range of hardware—both on-premises and in the cloud.
“With the Asylo toolset, Gemalto sees accelerated use of secure enclaves for high security assurance applications in cloud and container environments. Asylo makes it easy to attach container-based applications to securely isolate computations. Combining this with Gemalto’s SafeNet Data Protection On Demand paves the way to build trust across various industry applications, including; 5G, Virtual Network Functions (VNFs), Blockchain, payments, voting systems, secure analytics and others that require secure application secrets. Using Asylo, we envision our customers gaining deployment flexibility across multiple cloud environments and the assurance of meeting strict regulatory requirements for data protection and encryption key ownership.”
— Todd Moore, Senior Vice President of Data Protection at Gemalto
The Asylo framework allows developers to easily build applications and make them portable, so they can be deployed on a variety of software and hardware backends. With Asylo, we supply a Docker image via Google Container Registry that includes all the dependencies you need to run your container anywhere. This flexibility allows you to take advantage of various hardware architectures with TEE support without modifying your source code.

Asylo offers unique benefits over alternative approaches to confidential computing:
  • Ease of use. With Asylo, it’s easy to create apps that take advantage of the security properties of TEEs. You won’t need to learn a completely new programming model, or rewrite your app.
  • Portability and deployment flexibility. Asylo applications do not need to be aware of the intricacies of specific TEE implementations; you can port your apps across different enclave backends with no code changes. Your apps can run on your laptop, a workstation under your desk, a virtual machine in an on-premises server, or an instance in the cloud. We are exploring future backends based on AMD Secure Encryption Virtualization (SEV) technology, Intel® Software Guard Extensions (Intel® SGX), and other industry-leading hardware technologies that could support the same rebuild-and-run portability.
  • Open source. As an open-source framework, everyone can take advantage of confidential computing technology. Keep on the lookout for Asylo’s rapidly-evolving capabilities!

The Asylo roadmap

With Asylo, we can create the next generation of confidential computing applications together with the community. In version 0.2, Asylo offers an SDK and tools to help you develop portable enclave applications. Coming soon, Asylo will also allow you to run your existing applications in an enclave—just copy your app into the Asylo container, specify the backend, rebuild, and run!

We look forward to seeing how you use, build on, and extend Asylo. Your input and contributions will be critical to the success of the project and ensure Asylo grows to support your needs.

Getting started with Asylo

It’s easy to get started with Asylo—simply download the Asylo sources and pre-built container image from Google Container Registry. Be sure to check out the samples in the container, expand on them, or use them as a guide when building your own Asylo apps from scratch.

Check out our quick-start guide, read the documentation, and join our mailing list to take part in the discussion. We look forward to hearing from you on GitHub!

Exploring container security: Using Cloud Security Command Center (and five partner tools) to detect and manage an attack



Editor’s note: This is the sixth in a series of blog posts on container security at Google.

If you suspect that a container has been compromised, what do you do? In today’s blog post on container security, we’re focusing in on container runtime security—how to detect, respond to, and mitigate suspected threats for containers running in production. There’s no one way to respond to an attack, but there are best practices that you can follow, and in the event of a compromise, we want to make it easy for you to do the right thing.

Today, we’re excited to announce that you’ll soon be able to manage security alerts for your clusters in Cloud Security Command Center (Cloud SCC), a central place on Google Cloud Platform (GCP) to unify, analyze and view security data across your organization. Further, even though we just announced Cloud SCC a few weeks ago, already five container security companies have integrated their tools with Cloud SCC to help you better secure the containers you’re running on Google Kubernetes Engine.

With your Kubernetes Engine assets in Cloud SCC, you can view security alerts for your Kubernetes Engine clusters in a single pane of glass, and choose how to best take action. You’ll be able to view, organize and index your Kubernetes Engine cluster assets within each project and across all the projects that your organization is working on. In addition, you’ll be able to associate your container security findings to either specific clusters, container images and/or VM instances as appropriate.

Until then, let’s take a deeper look at runtime security in the context of containers and Kubernetes Engine.

Responding to bad behavior in your containers

Security operations typically includes several steps. For example, NIST’s well known framework includes steps to identify, protect, detect, respond, and recover. In containers, this translates to detecting abnormal behavior, remediating a potential threat, performing forensics after an incident, and enforcing runtime policies in isolated environments such as the new gVisor sandboxed container environment.

But first, how do you detect that a container is acting maliciously? Typically, this requires creating a baseline of what normal behaviour looks like, and using rules or machine learning to detect variation from that baseline. There are many ways to create that initial behavioral baseline (i.e., how a container should act), for example, using kprobes, tracepoints, and eBPF kernel inspection. Deviation from this baseline then triggers an alert or action.

If you do find a container that appears to be acting badly, there are several actions you might want to take, in increasing order of severity:

  • Just send an alert. This notifies your security response team that something unusual had been detected. For example, if security monitoring is relatively new in your environment, you might be worried about having too many false positives. Cloud SCC lets you unify container security signals with other security signals across your organization. With Cloud SCC, you can: see the live monitored state of container security issues in the dashboard; access the details either in the UI or via the API; and set up customer-defined filters to generate Cloud Pub/Sub topics that can then trigger email, SMS, or bugs in Jira.
  • Isolate a container. This moves the container to a new network, or otherwise restricts its network connectivity. For example, you might want to do this if you think one container is being used to perform a denial of service attack on other services.
  • Pause a container, e.g., `gcloud compute instances stop`. This suspends all running processes in the container. For example, if you detect suspected cryptomining, you might want to limit resource use and make a backup prior to further investigation.
  • Restart a container, e.g., `docker restart` or `kubectl delete pod`. This kills and restarts a running container, and resets the current state of the application. For example, if you suspect an attacker has created a foothold in your container, this might be a first step to counter their efforts, but this won’t stop an attacker from replicating an attack—just temporarily remove them.
  • Kill a container, i.e., `docker kill`. This kills a running container, halting all running processes (and less gracefully than `docker stop`). This is typically a last resort for a suspected malicious container.

Analyzing a security incident

After an incident, your security forensics team might step in to determine what really happened, and how they can prevent it the next time around. On Kubernetes Engine, you can look at a few different sources of event information:

  • Security event history and monitoring status in Cloud SCC. You can view the summary status of your assets and security findings in the dashboard, configure alerting and notification to a custom Cloud Pub/Sub topic and then query and explore specific events in detail either via the UI or API.
  • Container logs, kubelet logs, Docker logs, and audit logs in Stackdriver. Kubernetes Engine Audit Logging captures certain actions by default, both in the Kubernetes Engine API (e.g., create cluster, remove nodepool) and in the Kubernetes API (e.g., create a pod, update a DaemonSet).
  • Snapshots. You can snapshot a container’s filesystem in docker with `docker export`.

Announcing our container runtime security partners

To give you the best options for container runtime security on Google Cloud Platform, we’re excited to announce five partners who have already integrated with Cloud SCC: Aqua Security, Capsule8, Stackrox, Sysdig Secure, and Twistlock. These technical integrations allow you to use their cutting-edge security tools with your deployments, and view their findings and recommendations directly in Cloud SCC.

Aqua Security

Aqua’s integration with Cloud SCC provides real-time visibility into container security events and policy violations, including:

  • Inventory of vulnerabilities in container images in Google Container Registry, and alerts on new vulnerabilities
  • Container user security violations, such as privilege escalation attempts
  • Attempts to run unapproved images
  • Policy violations of container network, process, and host resource usage

To learn more and get a demo of Aqua’s integration with Google Cloud SCC, visit aquasec.com/gcp

Capsule8

Capsule8 is a real-time, zero-day attack detection platform purpose-built for modern production infrastructures. The Capsule8 integration with Google delivers continuous security across GCP environments to detect and help shut down attacks as they happen. Capsule8 runs entirely in the customer's Google Compute Engine environment and accounts and only requires a lightweight installation-free sensor running on each Compute Engine instance to stream behavioral telemetry to identify and help shut down zero-day attacks in real-time.

For more information on Capsule8’s integration with GCP, please visit: https://capsule8.com/capsule8-for-google-cloud-platform/

Stackrox

StackRox has partnered with Google Cloud to deliver comprehensive security for customers running containerized applications on Kubernetes Engine. StackRox visualizes the container attack surface, exposes malicious activity using machine learning, and stops attacks. Under the partnership, StackRox is working closely with the GCP team to offer an integrated experience for Kubernetes and Kubernetes Engine users as part of Cloud SCC.

“My current patchwork of security vendor solutions is no longer viable – or affordable – as our enterprise is growing too quickly and cyber threats evolve constantly. StackRox has already unified a handful of major product areas into a single security engine, so moving to containers means positive ROI."

- Gene Yoo, Senior Vice President and Head of Information Security at City National Bank

For more information on StackRox’s integration with GCP, please visit: https://www.stackrox.com/google-partnership

Sysdig Secure

By bringing together container visibility and a native Kubernetes Engine integration, Sysdig Secure provides the ability to block threats, enforce compliance, and audit activity across an infrastructure through microservices-aware security policies. Security events are enriched with hundreds of container and Kubernetes metadata before being sent to Cloud SCC. This brings the most relevant signals to your attention and correlates Sysdig events with other security information sources so you can have a single point of view and the ability to react accordingly at all levels.

"We chose to develop on Google Cloud for its robust, cost-effective platform. Sysdig is the perfect complement because it allows us to effectively secure and monitor our Kubernetes services with a single agent. We're excited to see that Google and Sysdig are deepening their partnership through this product integration.”

- Ashley Penny, VP of infrastructure, Cota Healthcare. 

For more information on Sysdig Secure’s integration with GCP, please visit: https://sysdig.com/gke-monitoring/

Twistlock

Twistlock surfaces cloud-native security intel vulnerability findings, compliance posture, runtime anomalies, and firewall logs directly into Cloud SCC. Customers can use Cloud SCC's big data capabilities to analyze and alert at scale, integrating container, serverless, and cloud-native VM security intelligence alongside other apps and workloads connected to Cloud SCC.

"Twistlock enables us to pinpoint vulnerabilities, block attacks, and easily enforce compliance across our environment, giving our team the visibility and control needed to run containers at scale."

- Anthony Scodary, Co-Founder of Gridspace

For more information on Twistlock’s integration with GCP, please visit: https://twistlock.com/partners/google-cloud-platform

Now you have the tools you need to protect your containers! Safe computing!

And if you’re at KubeCon in Copenhagen, join us at our booth for a demo and discussion around container security.

Open-sourcing gVisor, a sandboxed container runtime



Containers have revolutionized how we develop, package, and deploy applications. However, the system surface exposed to containers is broad enough that many security experts don't recommend them for running untrusted or potentially malicious applications.

A growing desire to run more heterogenous and less trusted workloads has created a new interest in sandboxed containers—containers that help provide a secure isolation boundary between the host OS and the application running inside the container.

To that end, we’d like to introduce gVisor, a new kind of sandbox that helps provide secure isolation for containers, while being more lightweight than a virtual machine (VM). gVisor integrates with Docker and Kubernetes, making it simple and easy to run sandboxed containers in production environments.

Traditional Linux containers are not sandboxes

Applications that run in traditional Linux containers access system resources in the same way that regular (non-containerized) applications do: by making system calls directly to the host kernel. The kernel runs in a privileged mode that allows it to interact with the necessary hardware and return results to the application.

With traditional containers, the kernel imposes some limits on the resources the application can access. These limits are implemented through the use of Linux cgroups and namespaces, but not all resources can be controlled via these mechanisms. Furthermore, even with these limits, the kernel still exposes a large surface area that malicious applications can attack directly.

Kernel features like seccomp filters can provide better isolation between the application and host kernel, but they require the user to create a predefined whitelist of system calls. In practice, it’s often difficult to know which system calls will be required by an application beforehand. Filters also provide little help when a vulnerability is discovered in a system call that your application requires.

Existing VM-based container technology

One approach to improve container isolation is to run each container in its own virtual machine (VM). This gives each container its own "machine," including kernel and virtualized devices, completely separate from the host. Even if there is a vulnerability in the guest, the hypervisor still isolates the host, as well as other applications/containers running on the host.

Running containers in distinct VMs provides great isolation, compatibility, and performance, but may also require a larger resource footprint.

Kata containers is an open-source project that uses stripped-down VMs to keep the resource footprint minimal and maximize performance for container isolation. Like gVisor, Kata contains an Open Container Initiative (OCI) runtime that is compatible with Docker and Kubernetes.

Sandboxed containers with gVisor

gVisor is more lightweight than a VM while maintaining a similar level of isolation. The core of gVisor is a kernel that runs as a normal, unprivileged process that supports most Linux system calls. This kernel is written in Go, which was chosen for its memory- and type-safety. Just like within a VM, an application running in a gVisor sandbox gets its own kernel and set of virtualized devices, distinct from the host and other sandboxes.

gVisor provides a strong isolation boundary by intercepting application system calls and acting as the guest kernel, all while running in user-space. Unlike a VM which requires a fixed set of resources on creation, gVisor can accommodate changing resources over time, as most normal Linux processes do. gVisor can be thought of as an extremely paravirtualized operating system with a flexible resource footprint and lower fixed cost than a full VM. However, this flexibility comes at the price of higher per-system call overhead and application compatibility—more on that below.

"Secure workloads are a priority for the industry. We are encouraged to see innovative approaches like gVisor and look forward to collaborating on specification clarifications and making improvements to joint technical components in order to bring additional security to the ecosystem."
— Samuel Ortiz, member of the Kata Technical Steering Committee and Principal Engineer at Intel Corporation
“Hyper is encouraged to see gVisor’s novel approach to container isolation. The industry requires a robust ecosystem of secure container technologies, and we look forward to collaborating on gVisor to help bring secure containers into the mainstream.”
— Xu Wang, member of the Kata Technical Steering Committee and CTO at Hyper.sh

Integrated with Docker and Kubernetes

The gVisor runtime integrates seamlessly with Docker and Kubernetes though runsc (short for "run Sandboxed Container"), which conforms to the OCI runtime API.

The runsc runtime is interchangeable with runc, Docker's default container runtime. Installation is simple; once installed it only takes a single additional flag to run a sandboxed container in Docker:

$ docker run --runtime=runsc hello-world
$ docker run --runtime=runsc -p 3306:3306 mysql

In Kubernetes, most resource isolation occurs at the pod level, making the pod a natural fit for a gVisor sandbox boundary. The Kubernetes community is currently formalizing the sandbox pod API, but experimental support is available today.

The runsc runtime can run sandboxed pods in a Kubernetes cluster through the use of either the cri-o or cri-containerd projects, which convert messages from the Kubelet into OCI runtime commands.

gVisor implements a large part of the Linux system API (200 system calls and counting), but not all. Some system calls and arguments are not currently supported, as are some parts of the /proc and /sys filesystems. As a result, not all applications will run inside gVisor, but many will run just fine, including Node.js, Java 8, MySQL, Jenkins, Apache, Redis, MongoDB, and many more.

Getting started

As developers, we want the best of both worlds: the ease of use and portability of containers, and the resource isolation of VMs. We think gVisor is a great step in that direction. Check out our repo on GitHub to find how to get started with gVisor and to learn more of the technical details behind it. And be sure to join our Google group to take part in the discussion!

If you’re at KubeCon in Copenhagen join us at our booth for a deep dive demo and discussion.



Also check out an interview with the gVisor PM to learn more.


Exploring container security: Protecting and defending your Kubernetes Engine network



Editor’s note: This is the fourth in a series of blog posts on container security at Google. You can find the other posts in the series here.

Security is a crucial factor in deciding which public cloud provider to move to—if at all. Containers have become the standard way to deploy applications both in the public cloud and on-premises, and Google Kubernetes Engine implements several best practices to ensure the security and privacy of your deployments. In this post, we’ll answer some of your questions related to container networking security of Kubernetes Engine, and how it differs from traditional VM networking security.

Kubernetes’ design greatly simplifies networking by abstracting management of load balancer and firewall configurations. In addition, Kubernetes Engine also offers a variety of network security tools to help you lock down your network:
  • Private Clusters gives your workloads private RFC 1918 IP addresses, taking them off the public internet. Authorized networks let you restrict access to your masters to a specific set of IP address ranges. 
  • Virtual Private Cloud (VPC) Networks provide private global connectivity between your resources, including Kubernetes Engine clusters, offering flexibility in managing your deployment with respect to IP address management, firewalls and routes. 
  • VPC service controls allow you to restrict the access to the data from trusted network sources and mitigate data exfiltration risks. 
  • Kubernetes Network Policies let you restrict pod to pod traffic in your Kubernetes cluster, giving you the ability to isolate your microservices from one another and from the outside world. 
  • Encryption for data in transit as it moves across a physical boundary is standard across Google Cloud Platform (GCP). 
  • Secure access for GCP services from your on-premises deployments using Private Google Access and secure connection to your GCP deployments using our advanced peering capabilities to allow no traffic to transit through the internet. 
  • Layer 7 Protection with support for HTTPS with Ingress on Kubernetes Engine. Protection against DDoS and other layer 4 attacks such as SYN floods through Google Cloud HTTP(S) and TCP load balancers.

Google Cloud offers a wealth of services and functionality to help you lock down the network connected to your Kubernetes Engine clusters. Let’s drill down into each of these offerings a bit further.

Isolate your workload from the public web using Private Clusters (beta) 

We recently announced the beta availability of Kubernetes Engine Private Clusters, allowing you to deploy clusters privately as part of your Google VPC and your nodes with only private RFC 1918 IP addresses. Private Clusters ensures network segmentation, reducing the scope and complexity of PCI-DSS compliance. Learn more about the specific segmentation requirements for PCI DSS.


Learn more about Private Clusters.

Restrict access to your master using Authorized Networks (beta) 

By default, Kubernetes Engine uses both Transport Layer Security (TLS) and IAM authentication to provide secure access to your master endpoint from the public internet. Authorized Networks give you additional protection by allowing a whitelisted range of IP addresses access to your master endpoint, blocking uncontrolled network access. Note that if you’re using a Private Cluster, your master endpoint will use Authorized Networks by default.

Run your Kubernetes Engine cluster in a global Virtual Private Cloud


Traditional VPCs are regional connected silos that must be stitched together through either the public internet or VPN, leading to a sprawl of connections to your on-premises deployment. Google’s VPC offers global connectivity for all your GCP workloads (e.g. Compute Engine VMs and Kubernetes clusters), removing the need for public internet routing or extra VPNs.

Restrict pod-to-pod traffic with a Kubernetes Network Policy


Kubernetes Network Policies allow you to restrict pod-to-pod traffic based on high-level policy specification. In Kubernetes Engine, these are supported using the Calico Network Plugin, created in collaboration with Tigera. Network policies help prevent an attacker from moving laterally. For example, using a network policy, a compromised front-end service can be restricted from communicating with a back-end service such as billing, giving you peace of mind through micro-segmentation of your services in the cloud. By restricting access to your services, this provides a building block to securing access and isolation in a multi-tenant microservices environment.
GCP also allows you to set up firewall rules between your Kubernetes Engine cluster nodes within your VPC network. These rules apply to egress traffic from instances and ingress traffic to instances in the network. Set up firewalls for your deployment today. Learn more about setting up network policies.

Encrypt data in transit when it leaves a physical boundary


Kubernetes Engine, as part of Google Cloud, employs several measures to ensure the authenticity, integrity and privacy of your data in transit. Kubernetes Engine protects communications between your nodes and the master through encryption, or to other services. Kubernetes Engine encrypts all data in transit at one or more network layers when data moves outside physical boundaries not controlled by Google Cloud or on behalf of Google Cloud. We use TLS to encrypt data in transit, and provide authentication, integrity and encryption using the HTTPS protocol with a certificate from a public certificate authority. Find out more in the Encryption in transit whitepaper.

Securely connect to other Google Cloud services and your GCP deployments


Private Google Access allows private RFC 1918 connectivity across VPC networks on a subnet level to other critical Google services such as Cloud Spanner, Cloud Pub/Sub, Cloud Dataproc and Cloud Storage. This allows you to securely connect to Google’s managed services without exposing your traffic to the public internet.
For connecting your on-premises infrastructure to GCP, your options include:
  • Dedicated Interconnect, which offers you enterprise grade connectivity from your on-premises network to your GCP VPC 
  • Google Cloud Platform VPN, lets you securely connect your on-premises cluster to your Kubernetes Engine deployment 

Secure your application traffic with Ingress and HTTPS 

For encrypted communications between the client to the load balancer, you can specify a TLS private key and certificate to be used by the Ingress controller. You can configure Ingress by specifying a secret or a GCP SSL certificate that contains a TLS private key and certificate. Learn how to use TLS with Ingress here.

Protect your deployment from DDoS attacks using our HTTP(S) and TCP load balancers


Our HTTP(S) and TCP load balancers are the result of expertise Google has gained running our services at scale. They scale up easily and offer protection against common attacks such as SYN floods, IP fragment floods, and port exhaustion. For more details, you can read the Best Practices for DDoS Protection and Mitigation on GCP whitepaper or watch the cloud networking blueprint for securing your workloads talk.

We hope that this gives you a good overview of the security features and techniques at your disposal when configuring network access to your Kubernetes Engine environment. Stay tuned next week for the next post in the container security series discussing how to harden your Kubernetes Engine environment.

Improving the Google Cloud Storage backend for HashiCorp Vault



HashiCorp Vault is a powerful open source tool for secrets management, popular with many Google Cloud Platform (GCP) customers today. HashiCorp Vault provides "secret management as a service," acting as a static secret store for encrypted key-value pairs; a secret generation tool to dynamically generate on-the-fly credentials; and pass-through encryption service so that applications don’t need to roll their own encryption. Today, we're announcing exciting improvements to the existing Google Cloud Storage backend for HashiCorp Vault, including high availability.

As mentioned in our blog post announcing Google Cloud Spanner as a supported HashiCorp Vault storage backend, we strive to make Google Cloud an excellent platform on which to operationalize Vault for all users and use cases. Your feedback from the Cloud Spanner integration was overwhelmingly positive, but many of you are already leveraging the community-supported Cloud Storage backend and don’t want to migrate your existing data to a different storage system. GCP’s wealth of offerings let you choose the best storage options to meet your needs, and now you can choose from both Cloud Spanner and Cloud Storage for HashiCorp Vault storage backends.

The improved Cloud Storage HashiCorp Vault storage backend is completely backwards compatible with the existing solution, but includes a number of new features and benefits:
  • High availability - In addition to Cloud Storage's built-in multi-region architecture, the improved HashiCorp Vault storage backend also supports running Vault in "high availability" mode. By default, HashiCorp Vault runs as a single tenant, relying on the storage backend to provide distributed locking and leader election. By leveraging object metadata for read-modify-write conditions in Cloud Storage, the improved storage backend allows for a highly available Vault cluster with just a single line of configuration. You can read more about HashiCorp Vault's High Availability model in the documentation.
  • Support for default application credentials - Previously the Cloud Storage Vault storage backend required you to create a dedicated service account and credentials file. While you can still specify a credentials file, the storage backend now supports pulling default application credentials, such as those from your local gcloud installation or Application Default Credentials if you're running Vault on GCP.
  • Enterprise-grade security - Cloud Storage follows the same security best practices as other Google products. Objects stored in Cloud Storage are encrypted by default, and it uses IAM to provide granular permission management on buckets and folders. Google’s infrastructure has many security differentiators, including secure boot using Google’s custom-designed security chip Titan, and Google’s private network backbone.


Getting started


To get started, download and install the latest version of HashiCorp Vault. The improvements to the Cloud Storage backend for Vault, including high availability mode, were added in Vault 0.10 (released on April 10, 2018). Please ensure you're running Vault 0.10 or later before continuing.

Next, create a Cloud Storage bucket using the gsutil CLI tool (part of the gcloud CLI) to store the Vault data . You can also create the bucket using the web interface or API directly:

$ gsutil mb -c regional -l us-east4 gs://company-vault-data

In this example, we created a bucket named "company-vault-data." Note that Cloud Storage bucket names must be globally unique.

Next, create a Vault configuration file configured to use Cloud Storage as the storage backend:

# config.hcl
storage "gcs" {
  bucket = "company-vault-data"
}

Start Vault with the configuration file. Note that this example uses Vault's built-in development mode, which does not represent best practices or a production installation, but it's the fastest way to try the improved Cloud Storage storage backend for HashiCorp Vault. For more details on a production-grade Vault installation, please read the Vault production hardening guide.

$ export VAULT_ADDR=http://127.0.0.1:8200
$ sudo vault server -dev -config=config.hcl

During this process, Vault authenticates and connects to Cloud Storage to populate and manage objects in the provided bucket. After a few seconds, you can view the objects in the web interface and see that data has been populated.

You can now create, read, update and delete secrets:

$ vault kv write secret/my-secret foo=bar

To learn more about the backend configuration options, read the HashiCorp Vault Cloud Storage backend documentation. To learn more about Cloud Storage, check out the documentation.

Toward a seamless Vault experience on GCP


With the Cloud Spanner and Cloud Storage Vault storage backends, Vault users can choose which Google-supported storage backend is best for them. In addition to supporting our customers, we are delighted to continue our long-standing relationship with HashiCorp as part of our ongoing partnership. Be sure to follow us on Twitter and open a GitHub issue if you have any questions.

Exploring container security: Digging into Grafeas container image metadata



Editor’s note: This is the third in a series of blog posts on container security at Google.

The great thing about containers is how easy they are to create, modify and share. But that also raises the question of whether or not they're safe to deploy to production. One way to answer that is to track metadata about your container, for example, who worked on it, where it's stored, and whether it has any known vulnerabilities.

Last October, Google and several partners announced Grafeas, an open-source project that provides a structured metadata API for container images. Grafeas simplifies how you handle metadata, including security concerns like package vulnerability scan results, and keeps track of lots of different kinds of information:

  • "Build" metadata can be used to certify that a container image was built according to your build policies using a trusted builder with checked-in source code. 
  • "Package Vulnerability" metadata contains information from a vulnerability scanning service and lets you create policies on deploying vulnerable containers. 
  • "Image Basis" metadata contains information about the base image from which your container was derived as well as additional layers used in building the container image. 
  • "Package Manager" metadata indicates which packages were installed in the container image. 
  • "Deployment History" metadata allows you to track which resources were deployed and when.

Tracking Grafeas’ metadata lets you know what containers are in your environment, and also to enforce restrictions on which containers get deployed. You should review Grafeas metadata for compliance with your policies before deploying. This is where one last type of Grafeas metadata comes in—the mighty "Attestation." If the metadata you pull from Grafeas is consistent with your policy, you can then write an attestation that certifies that your image complies with your deployment requirements. Then, using the Kubernetes Admission Controller, you can check for the expected attestations, and block deployment when they aren’t present.

occurrencesResponse := grafeas.ListOccurrencesResponse{}

 imageUrl := "https://gcr.io/myproject/name@sha256hash"
 hasFixableVulnerability := false
// Iterate through occurrences looking for vulnerability occurrences for this image
for _, occurrence := range occurrencesResponse.Occurrences {
  resourceUrl := occurrence.ResourceUrl
  if imageUrl != resourceUrl && occurrence.Kind != grafeas.Note_PACKAGE_VULNERABILITY {
     continue
  }
  details := occurrence.GetVulnerabilityDetails()
  issues := details.GetPackageIssue()
  // This this vulnerability is fixable, we should not insert an attestation
  for _, i := range issues {
     if i.FixedLocation != nil {
        hasFixableVulnerability = true
     }
  }
}
// If there are no fixible vulnerabilities, create an attestation occurrence
if !hasFixableVulnerability {
  occ := &grafeas.Occurrence{
     ResourceUrl: imageUrl,
     NoteName:    "projects/vulnerabilty-certification/notes/vuln-free",
     Details: &grafeas.Occurrence_AttestationDetails{
        AttestationDetails: &grafeas.AttestationAuthority_AttestationDetails{
           Signature: &grafeas.AttestationAuthority_AttestationDetails_PgpSignedAttestation{
              PgpSignedAttestation: &grafeas.PgpSignedAttestation{
                 Signature: "MySignature",
                 ContentType: grafeas.PgpSignedAttestation_SIMPLE_SIGNING_JSON,
                 KeyId: "ThisIsMyKeyId"

              },
           },

           },
        },
     }
     req := grafeas.CreateOccurrenceRequest{Occurrence:occ}

  }

In fact, you can use Grafeas to enforce all kinds of security policies. Check out the tutorial by Kelsey Hightower on how to use Grafeas to allow only container images signed by a specific private key. But it doesn’t stop there. For example, you can write policies to block container images with vulnerabilities from being deployed, ensure that all your deployed images are built from a base image sanctioned by your security team, or require images to go through your build pipeline. The possibilities are endless!

How Shopify uses Grafeas to manage metadata for its 500k container images


Big changes in infrastructure, like moving to Kubernetes for container orchestration, can create opportunities to change the way you govern your workloads. Here at Shopify, we use Grafeas as the central place to store all the metadata about our deploys, helping us build six thousand containers a day, and making our lives easier.

Grafeas lets us answer fundamental questions like:

When was the container created? What code runs inside it? Who created it? Where is the running container located? Is the container vulnerable to a known exploit and do we need to pull it out of production to replace it?

Shopify’s Continuous Integration/Deployment (CI/CD) pipeline supplies information about who built the code, when it was built, what the commit hash is, and whether or not tests passed. We store this data in the “build” metadata in Grafeas, leaving a digital paper trail that can be audited to track changes.

After build, we test the container for known vulnerabilities. Since that list of vulnerabilities can change at any time, we check our images at regular intervals. The information in Grafeas’ “package vulnerability” metadata field is updated at each check and allows us to make informed decisions about when to pull an image out of production and deploy a new secure one.

Next up, we’re working on how to track information about deploys in the deployment history metadata field, so that we know when, where and by whom a container was deployed, as well as its current status. We will use that information to create a “bill of materials” of what’s running in our environment.

The metadata in Grafeas gives us an answer to all of our questions around deploys. We also use it to restrict the deployment of containers. During the creation of a workload, the admission controller makes its decision by querying Grafeas for the container’s metadata  if it cannot find anything that violates predefined policies, for example the existence of known vulnerabilities, it allows the workload to be created.

Using Grafeas in production gives us a 360-degree view of our deploys so we can protect our infrastructure and maintain a history of what’s going on in our cloud.

Get involved in Grafeas 


We’re currently working toward a beta version of the Grafeas API spec. In addition, Grafeas is also working with other projects, such as the in-toto team, to allow in-toto to create attestations based on Grafeas metadata.

If you’d like to contribute, Grafeas is always looking for more contributors as we grow this community. Join us!

How to automatically scan Cloud Storage buckets for sensitive data: Taking charge of your security



Security in the cloud is often a matter of identifying—and sticking to—some simple best practices. A few months ago, we discussed some steps you can take to harden the security of your Cloud Storage buckets. We covered how to set up proper access permissions and provided tips on using tools like the Data Loss Prevention API to monitor your buckets for sensitive data. Today, let’s talk about how to automate data classification using the DLP API and Cloud Functions, Google Cloud Platform’s event-driven serverless compute platform that makes it easy for you to integrate and extend cloud services with code.

Imagine you need to regularly share data with a partner outside of your company, and this data cannot contain any sensitive elements such as Personally Identifiable Information (PII). You could just create a bucket, upload data to it, and grant access to your partner, but what if someone uploads the wrong file or doesn’t know that they aren’t supposed to upload PII? With the DLP API and Cloud Functions, you can automatically scan this data before it’s uploaded to the shared storage bucket.

Setting this up is easy: Simply create three buckets—one in which to upload data, one to share and one for any sensitive data that gets flagged. Then:

  1. Configure access appropriately so that relevant users can put data in the “upload” bucket 
  2. Write a Cloud Function triggered by an upload that scans the data using the DLP API 
  3. Based on any DLP findings, automatically move data into the share bucket or into a restricted bucket for further review.

To get you started, here’s a tutorial with detailed instructions including the Cloud Functions script. You can get it up and running in just a few minutes from the Cloud Console or via Cloud Shell. You can then easily modify the script for your environment, and add more advanced actions such as sending notification emails, creating a redacted copy or triggering approval workflows.

We hope we’ve showed you some proactive steps you can take to prevent sensitive data from getting into the wrong hands. To learn more, check out the documentation for the DLP API and Cloud Functions.

Best practices for securing your Google Cloud databases



If information is gold, the database is a treasure chest. Web applications store their most valuable data in a database, and lots of sites would cease to exist if their data were stolen or deleted. This post aims to give you a series of best practices to help protect and defend the databases you host on Google Cloud Platform (GCP).

Database security starts before the first record is ever stored. You must consider the impact on security as you design the hosting environment. From firewall rules to logging, there's a lot to think about both inside and outside the database.

First considerations


When it comes to database security, one of the fundamental things to consider is whether to deploy your own database servers or to use one of Google's managed storage services. This decision should be influenced heavily by your existing architecture, staff skills, policies and need for specialized functionality.

This post is not intended to sell you specific GCP products, but absent any overwhelming reasons to host your own database, we recommend using a managed version thereof. Managed database services are designed to work at Google scale, with all of the benefits of our security model. Organizations seeking compliance with PCI, SOX, HIPAA, GDPR, and other compliance regimes will appreciate the significant reduction in effort with a shared responsibility model. And even if these rules and regulations don't apply to your organization, I recommend following the PCI SAQ A (payment card industry self-assessment questionnaire type A) as a baseline set of best practices.

Access controls


You should limit access to the database itself as much as possible. Self-hosted databases should have VPC firewall rules locked down to only allow ingress from and egress to authorized hosts. All ports and endpoints not specifically required should be blocked. If possible, ensure changes to the firewall are logged and alerts are configured for unexpected changes. This happens automatically for firewall changes in GCP. Tools like Forseti Security can monitor and manage security configurations for both Google managed services and custom databases hosted on Google Compute Engine instances.

As you prepare to launch your database, you should also consider the environment in which it operates. Service accounts streamline authorization to Google databases using automatically rotating keys, and you can manage rotation for self-hosted databases in GCP using Cloud Key Management System (KMS).


Data security


Always keep your data retention policy in mind as you implement your schema. Sensitive data that you have no use for is a liability and should be archived or pruned. Many compliance regulations provide specific guidance (HIPAA, PCI, GDPR, SOX) to identify that data. You may find it helpful to operate under the pessimistic security model in which you assume your application will be cracked and your database will be exfiltrated. This can help clarify some of the decisions you need to make regarding retention, encryption at rest, etc.

Should the worst happen and your database is compromised, you should receive alerts about unexpected behavior such as spikes in egress traffic. Your organization may also benefit from using "canary" data—specially crafted information that should never be seen in the real world by your application under normal circumstances. Your application should be designed to detect a canary account logging in or canary values transiting the network. If found, your application should send you alerts and take immediate action to stem the possible compromise. In a way, canary data is similar to retail store anti-theft tags. These security devices have known detection characteristics and can be hidden inside a product. A security-conscious retailer will set up sensors at their store exists to detect unauthorized removal of inventory.

Of course, you should develop and test a disaster recovery plan. Having a copy of the database protects you from some failures, but it won't protect you if your data has been altered or deleted. A good disaster recovery plan will cover you in the event of data loss, hardware issues, network availability and any other disaster you might expect. And as always, you must regularly test and monitor the backup system to ensure reliable disaster recovery.

Configuration


If your database was deployed with a default login, you should make it a top priority to change or disable that account. Further, if any of your database accounts are password-protected, make sure those passwords are long and complex; don’t use simple or empty passwords used under any circumstances. If you're able to use Cloud KMS, that should be your first choice. Beyond that, be sure to develop a schedule for credential rotation and define criteria for out-of-cycle rekeying.

Regardless of which method you use for authentication, you should have different credentials for read, write and admin-level privileges. Even if an application performs both read and write operations, separate credentials can limit the damage caused by bad code or unauthorized access.

Everyone who needs to access the database should have their own private credentials. Create service accounts for each discrete application with only the permissions required for that service. Cloud Identity and Access Management is a powerful tool for managing user privileges in GCP; generic administrator accounts should be avoided as they mask the identity of the user. User credentials should restrict rights to the minimum required to perform their duties. For example, a user that creates ad-hoc reports should not be able to alter schema. Consider using views, stored procedures and granular permissions to further restrict access to only what a user needs to know and further mitigate SQL injection vulnerabilities.

Logging is a critical part of any application. Databases should produce logs for all key events, especially login attempts and administrative actions. These logs should be aggregated in an immutable logging service hosted apart from the database. Whether you're using Stackdriver or some other service, credentials and read access to the logs should be completely separate from the database credentials.


Whenever possible, you should implement a monitor or proxy that can detect and block brute force login attempts. If you’re using Stackdriver, you could set up an alerting policy to a Cloud Function webhook that keeps track of recent attempts and creates a firewall rule to block potential abuse.

The database should run as an application-specific user, not root or admin. Any host files should be secured with appropriate file permissions and ownership to prevent unauthorized execution or alteration. POSIX compliant operating systems offer chown and chmod to set file permissions, and Windows servers offer several tools as well. Tools such as Ubuntu's AppArmor go even further to confine applications to a limited set of resources.


Application considerations


When designing application, an important best practice is to only employ encrypted connections to the database, which eliminates the possibility of an inadvertent leak of data or credentials via network sniffing. Cloud SQL users can do this using Google's encrypting SQL proxy. Some encryption methods also allow your application to authenticate the database host to reduce the threat of impersonation or man-in-the-middle attacks.
If your application deals with particularly sensitive information, you should consider whether you actually need to retain all of its data in the first place. In some cases, handling this data is governed by a compliance regime and the decision is made for you. Even so, additional controls may be prudent to ensure data security. You may want to encrypt some data at the application level in addition to automatic at-rest encryption. You might reduce other data to a hash if all you need to know is whether the same data is presented again in the future. If using your data to train machine learning models, consider reading this article on managing sensitive data in machine learning.

Application security can be used to enhance database security, but must not be used in place of database security. Safeguards such as sanitization must be in place for any input sent to the database, whether it’s data for storage or parameters for querying. All application code should be peer-reviewed. Security scans for common vulnerabilities, including SQL injection and XSS, should be automated and run frequently.

The computers of anyone with access rights to the database should be subject to an organizational security policy. Some of the most audacious breaks in security history were due to malware, lax updates or mishandled portable data storage devices. Once a workstation is infected, every other system it touches is suspect. This also applies to printers, copiers and any other connected device.

Do not allow unsanitized production data to be used in development or test environments under any circumstances. This one policy will not only increase database security, but will all but eliminate the possibility of a non-production environment inadvertently emailing customers, charging accounts, changing states, etc.

Self-hosted database concerns


While Google's shared responsibility model allows managed database users to relieve themselves of some security concerns, we can't offer equally comprehensive controls for databases that you host yourself. When self-hosting, it's incumbent on the database administrator to ensure every attack vector is secured.

If you’re running your own database, make sure the service is running on its own host(s), with no other significant application functions allowed on it. The database should certainly not share a logical host with a web host or other web-accessible services. If this isn’t possible, the service and databases should reside in a restricted network shared and accessible by a minimum number of other hosts. A logical host is the lowest level computer or virtual computer in which an application itself is running. In GCP, this may be a container or virtual machine instance, while on-premises, logical hosts can be physical servers, VMs or containers.
A common use case for running your own database is to replicate across a hybrid cloud. For example, a hybrid cloud database deployment may have a master and one or more replicas in the cloud, with a replica on-premises. In that event, don’t connect the database to the corporate LAN or other broadly available networks. Similarly, on-premises hardware should be physically secure so it can't be stolen or tampered with. A proper physical security regime employs physical access controls, automated physical access logs and remote video monitoring.

Self-hosted databases also require your regular attention, to make sure they’ve been updated. Craft a software update policy and set up regular alerts for out-of-date packages. Consider using a system management (Ubuntu, Red Hat) or configuration management tool that lets you easily perform actions in bulk across all your instances, monitor and upgrade package versions and configure new instances. Be sure to monitor for out-of-cycle changes to key parts of the filesystem such as directories that contain configuration and executable files. 

Several compliance regimes recommend an intrusion detection system or IDS. The basic function of an IDS is to monitor and react to unauthorized system access. There are many products available that run either on each individual server or on a dedicated gateway host. Your choice in IDS may be influenced by several factors unique to your organization or application. An IDS may also be able to serve as a monitor for canary data, a security tactic described above.

All databases have specific controls that you must adjust to harden the service. You should always start with articles written by your database maker for software-specific advice on hardening the server. Hardening guides for several popular databases are linked in the further reading section below.
The underlying operating system should also be hardened as much as possible, and all applications that are not critical for database function should be disabled. You can achieve further isolation by sandboxing or containerizing the database. Use articles written by your OS maker for variant-specific advice on how to harden the platform. Guides for the most common operating systems available in Compute Engine are linked in the further reading section below.

Organizational security 


Staff policies to enforce security is an important but often overlooked part of IT security. It's a very nuanced and deep topic, but here are a few general tips that will aid in securing your database:

All staff with access to sensitive data should be considered for a criminal background check. Insist on strict adherence to a policy of eliminating or locking user accounts immediately upon transfer or termination. Human account password policies should follow the 2017 NIST Digital Identity Guidelines, and consider running social engineering penetration tests and training to reduce the chance of staff inadvertently enabling an attack.

Further reading 


Security is a journey, not a destination. Even after you've tightened security on your database, application, and hosting environment, you must remain vigilant of emerging threats. In particular, self-hosted DBs come with additional responsibilities that you must tend to. For your convenience, here are some OS- and database-specific resources that you may find useful.

How to dynamically generate GCP IAM credentials with a new HashiCorp Vault secrets engine



Applications often require secrets such as credentials at build- or run-time. These credentials are an assertion of a service’s or user’s identity that they can use to authenticate to other services. On Google Cloud Platform (GCP), you can manage services or temporary users using Cloud Identity and Access Management (IAM) service accounts, which are identities whose credentials your application code can use to access other GCP services. You can access a service account from code running on GCP, in your on-premises environment, or even another cloud.

Protecting service account keys is critical—you should tightly control access to them, rotate them, and make sure they're not committed in code. But managing these credentials shouldn’t get harder as your number of services increases. HashiCorp Vault is a popular open source tool for secret management that allows users to store, manage and control access to tokens, passwords, certificates, API keys and many other secrets. Vault supports pluggable mechanisms known as secrets engines for managing different secret types. These engines allow developers to store, rotate and dynamically generate many kinds of secrets in Vault. Because Vault's secrets engines are pluggable, they each provide different functionality. Some engines store static data, others manage PKI and certificates, and others manage database credentials.

Today, we're pleased to announce a Google Cloud Platform IAM secrets engine for HashiCorp Vault. This allows a user to dynamically generate IAM service account credentials with which a human, machine or application can access specific Google Cloud resources using their own identity. To limit the impact of a security incident, Vault allows credentials to be easily revoked.

This helps address some common use cases, for example:
  • Restricting application access for recurring jobs: An application runs a monthly batch job. Instead of a hard-coded, long-lived credential, the application can request a short-lived GCP IAM credential at the start of the job. After a short time, the credential automatically expires, reducing the surface area for a potential attack.
  • Restricting user access for temporary users: A contracting firm needs 90 days of read-only access to build dashboards. Instead of someone generating this credential and distributing it to the firm, the firm requests this credential through Vault. This creates a 1-1 mapping of the credential to its users in audit and access logs.

Getting started with the IAM service account secret engine


Let’s go through an example for generating new service account credentials using Vault. This example assumes the Vault server is already up and running.

You can also watch a demo of the backend in our new video below.
First, enable the secrets engine:

$ vault secrets enable gcp 
Then, set up the engine with initial config and role sets:

$ vault write gcp/config \
credentials=@path/to/creds.json \
 ttl=3600 \
 max_ttl=86400

This config supplies default credentials that Vault will use to generate the service account keys and access tokens, as well as TTL metadata for the leases Vault assigns to these secrets when generated.

Role sets define the sets of IAM roles, bound to specific resources, that you assign to generated credentials. Each role set can generate one of two types of secrets: either `access_token` for one-use OAuth access tokens or `service_account_key` for long-lived service account keys. Here are some examples for both types of rolesets:

# Create role sets
$ vault write gcp/roleset/token-role-set \
    project="myproject" \
    secret_type="access_token" \
    bindings=@token_bindings.hcl
    token_scopes="https://www.googleapis.com/auth/cloud-platform"

$ vault write gcp/roleset/key-role-set \
    project="myproject" \
    secret_type="service_account_key"
    bindings=”
The above bindings param expects a string (or, using the special Vault syntax ‘@’, a path to a file containing this string) with the following HCL (or JSON) format
resource "path/to/my/resource" {
    roles = [
      "roles/viewer",
      "roles/my-other-role",
    ]
}

resource "path/to/another/resource" {
    roles = [
      "roles/editor",
    ]
}
Creating a new role set generates a new service account for a role set. When a user generates a set of credentials, they specify a role set (and thus service account) under which to create the credentials.

Once you have set up the secrets engine, a Vault client can easily generate new secrets:

$ vault read gcp/key/key-role-set

Key                 Value
---                 -----
lease_id            gcp/key/key-roleset/
lease_duration      1h
lease_renewable     true
key_algorithm       KEY_ALG_RSA_2048
key_type            TYPE_GOOGLE_CREDENTIALS_FILE
private_key_data    


$ vault read gcp/token/token-role-set

Key                 Value
---                 -----
lease_id           gcp/token/test/
lease_duration     59m59s
lease_renewable    false
token              ya29.c.restoftoken...
These credentials can then be used to make calls to GCP APIs as needed and can be automatically revoked by Vault.

To learn more, check out the GCP IAM service account secret engine documentation.

How WePay uses HashiCorp Vault on GCP


WePay is an online payment service provider who uses HashiCorp Vault on GCP. It currently runs HashiCorp Vault servers as virtual machines on Google Compute Engine for two primary use cases:
  • Plain vanilla secret storage using a configuration service: WePay has a service-oriented architecture built on Google Kubernetes Engine. Each microservice stores secrets such as passwords, tokens, private keys, and certificates in a centralized configuration service. This in turn uses the generic "kv" (key value) HashiCorp Vault secrets engine to manage application secrets. The configuration service authenticates services that talk to it, and authorizes those services to access their secrets at deployment time. Secrets are segmented by service using base paths, i.e., superSecurePaymentSystem would only be able to access secrets in the superSecurePaymentSystem path.
  • Key management using a key management service: WePay needs a way to centrally manage the provisioning, deprovisioning and rotation of encryption keys used in its applications. A central key management service generates encryption keys, and stores these in HashiCorp Vault using the "kv" secret engine.
WePay has its complete infrastructure built in GCP, and the introduction of the GCP IAM service account secrets engine will help to put in stronger security practices. WePay is exploring options on how to use the GCP IAM service account secrets engine in its infrastructure, and is excited by the possibilities.

Continuing work for HashiCorp Vault on GCP


We're excited to see what amazing applications and services users will build using the new HashiCorp Vault GCP secrets engine. This feature release is part of our long-standing ongoing partnership with HashiCorp since 2013. We're excited to continue working together to help HashiCorp users make the best use of GCP services and features. To get you up and running with, HashiCorp Vault for IAM service accounts, check out our solution brief “Using Vault on Compute Engine for Secret Management” for an overview of best practices, and a new video on authentication options.

As always, both HashiCorp and Google welcome contributions from the open-source community. Give us a tweet or open an issue on GitHub if you have any questions!

How to dynamically generate GCP IAM credentials with a new HashiCorp Vault secrets engine



Applications often require secrets such as credentials at build- or run-time. These credentials are an assertion of a service’s or user’s identity that they can use to authenticate to other services. On Google Cloud Platform (GCP), you can manage services or temporary users using Cloud Identity and Access Management (IAM) service accounts, which are identities whose credentials your application code can use to access other GCP services. You can access a service account from code running on GCP, in your on-premises environment, or even another cloud.

Protecting service account keys is critical—you should tightly control access to them, rotate them, and make sure they're not committed in code. But managing these credentials shouldn’t get harder as your number of services increases. HashiCorp Vault is a popular open source tool for secret management that allows users to store, manage and control access to tokens, passwords, certificates, API keys and many other secrets. Vault supports pluggable mechanisms known as secrets engines for managing different secret types. These engines allow developers to store, rotate and dynamically generate many kinds of secrets in Vault. Because Vault's secrets engines are pluggable, they each provide different functionality. Some engines store static data, others manage PKI and certificates, and others manage database credentials.

Today, we're pleased to announce a Google Cloud Platform IAM secrets engine for HashiCorp Vault. This allows a user to dynamically generate IAM service account credentials with which a human, machine or application can access specific Google Cloud resources using their own identity. To limit the impact of a security incident, Vault allows credentials to be easily revoked.

This helps address some common use cases, for example:
  • Restricting application access for recurring jobs: An application runs a monthly batch job. Instead of a hard-coded, long-lived credential, the application can request a short-lived GCP IAM credential at the start of the job. After a short time, the credential automatically expires, reducing the surface area for a potential attack.
  • Restricting user access for temporary users: A contracting firm needs 90 days of read-only access to build dashboards. Instead of someone generating this credential and distributing it to the firm, the firm requests this credential through Vault. This creates a 1-1 mapping of the credential to its users in audit and access logs.

Getting started with the IAM service account secret engine


Let’s go through an example for generating new service account credentials using Vault. This example assumes the Vault server is already up and running.

You can also watch a demo of the backend in our new video below.
First, enable the secrets engine:

$ vault secrets enable gcp 
Then, set up the engine with initial config and role sets:

$ vault write gcp/config \
credentials=@path/to/creds.json \
 ttl=3600 \
 max_ttl=86400

This config supplies default credentials that Vault will use to generate the service account keys and access tokens, as well as TTL metadata for the leases Vault assigns to these secrets when generated.

Role sets define the sets of IAM roles, bound to specific resources, that you assign to generated credentials. Each role set can generate one of two types of secrets: either `access_token` for one-use OAuth access tokens or `service_account_key` for long-lived service account keys. Here are some examples for both types of rolesets:

# Create role sets
$ vault write gcp/roleset/token-role-set \
    project="myproject" \
    secret_type="access_token" \
    bindings=@token_bindings.hcl
    token_scopes="https://www.googleapis.com/auth/cloud-platform"

$ vault write gcp/roleset/key-role-set \
    project="myproject" \
    secret_type="service_account_key"
    bindings=”
The above bindings param expects a string (or, using the special Vault syntax ‘@’, a path to a file containing this string) with the following HCL (or JSON) format
resource "path/to/my/resource" {
    roles = [
      "roles/viewer",
      "roles/my-other-role",
    ]
}

resource "path/to/another/resource" {
    roles = [
      "roles/editor",
    ]
}
Creating a new role set generates a new service account for a role set. When a user generates a set of credentials, they specify a role set (and thus service account) under which to create the credentials.

Once you have set up the secrets engine, a Vault client can easily generate new secrets:

$ vault read gcp/key/key-role-set

Key                 Value
---                 -----
lease_id            gcp/key/key-roleset/
lease_duration      1h
lease_renewable     true
key_algorithm       KEY_ALG_RSA_2048
key_type            TYPE_GOOGLE_CREDENTIALS_FILE
private_key_data    


$ vault read gcp/token/token-role-set

Key                 Value
---                 -----
lease_id           gcp/token/test/
lease_duration     59m59s
lease_renewable    false
token              ya29.c.restoftoken...
These credentials can then be used to make calls to GCP APIs as needed and can be automatically revoked by Vault.

To learn more, check out the GCP IAM service account secret engine documentation.

How WePay uses HashiCorp Vault on GCP


WePay is an online payment service provider who uses HashiCorp Vault on GCP. It currently runs HashiCorp Vault servers as virtual machines on Google Compute Engine for two primary use cases:
  • Plain vanilla secret storage using a configuration service: WePay has a service-oriented architecture built on Google Kubernetes Engine. Each microservice stores secrets such as passwords, tokens, private keys, and certificates in a centralized configuration service. This in turn uses the generic "kv" (key value) HashiCorp Vault secrets engine to manage application secrets. The configuration service authenticates services that talk to it, and authorizes those services to access their secrets at deployment time. Secrets are segmented by service using base paths, i.e., superSecurePaymentSystem would only be able to access secrets in the superSecurePaymentSystem path.
  • Key management using a key management service: WePay needs a way to centrally manage the provisioning, deprovisioning and rotation of encryption keys used in its applications. A central key management service generates encryption keys, and stores these in HashiCorp Vault using the "kv" secret engine.
WePay has its complete infrastructure built in GCP, and the introduction of the GCP IAM service account secrets engine will help to put in stronger security practices. WePay is exploring options on how to use the GCP IAM service account secrets engine in its infrastructure, and is excited by the possibilities.

Continuing work for HashiCorp Vault on GCP


We're excited to see what amazing applications and services users will build using the new HashiCorp Vault GCP secrets engine. This feature release is part of our long-standing ongoing partnership with HashiCorp since 2013. We're excited to continue working together to help HashiCorp users make the best use of GCP services and features. To get you up and running with, HashiCorp Vault for IAM service accounts, check out our solution brief “Using Vault on Compute Engine for Secret Management” for an overview of best practices, and a new video on authentication options.

As always, both HashiCorp and Google welcome contributions from the open-source community. Give us a tweet or open an issue on GitHub if you have any questions!