Tag Archives: Partners

Announcing Google Cloud Spanner as a Vault storage backend

By Seth Vargo, Staff Developer Advocate

HashiCorp Vault is a powerful open source tool for secrets management, popular with many Google Cloud Platform (GCP) customers today. Vault provides "secret management as a service," acting as a static secret store for encrypted key-value pairs; a secret generation tool to dynamically generate on-the-fly credentials; and pass-through encryption service so applications do not need to roll their own encryption. We strive to make Google Cloud an excellent platform on which to operationalize Vault.

Using Vault will require up-front configuration choices such as which Vault storage backend to use for data persistence. Some storage backends support high availability while others are single tenant; some operate entirely in your own datacenter while others require outbound connections to third-party services; some require operational knowledge of the underlying technologies, while others work without configuration. These options required you to consider tradeoffs across issues such as consistency, availability, scalability, replication, operationalization and institutional knowledge . . . until now.

Today we're pleased to announce Cloud Spanner as a storage backend for HashiCorp Vault. Building on the scalability and consistency of Google Cloud Spanner, Vault users gain all the benefits of a traditional relational database, the scalability of a globally-distributed data store and the availability (99.999% SLA for multi-region configurations) of a fully managed service.

With support for high-performance transactions and global consistency, using Cloud Spanner as a Vault storage backend brings a number of features and benefits:
  • High availability - In addition to Cloud Spanner's built-in high availability for data persistence, the Vault storage backend also supports running Vault in high availability mode. By default, Vault runs as a single tenant, relying on the storage backend to provide distributed locking and leader election. Cloud Spanner's global distribution and strong, consistent transactions allow for a highly available Vault cluster with just a single line of configuration.
  • Transactional support - Vault backends optionally support batch transactions for update and delete operations. Without transactional support, large operations—such as deleting an entire prefix or bootstrapping a cluster—can result in hundreds of requests. This can bottleneck the system or overload the underlying storage backend. The Cloud Spanner Vault storage backend supports Vault's transactional interface, meaning it collects a batch of related update/delete operations and issues a single API call to Spanner. Not only does this reduce the number of HTTP requests and networking overhead, but it also ensures a much speedier experience for bulk operations.
  • Enterprise-grade security - Cloud Spanner follows the same security best practices as other Google products. Data stored at rest in Cloud Spanner is encrypted by default, and Cloud Spanner uses IAM to provide granular permission management. Google’s infrastructure has many security differentiators, including backing by Google’s custom-designed security chip Titan, and Google’s private network backbone.
  • Google supported - This backend was designed and developed by Google developers, and is available through the Google open-source program. It's open for collaboration to the entire Vault community with the added benefit of support from the Google engineering teams.

Getting started


To get started, download and install the latest version of HashiCorp Vault. The Google Cloud Spanner Vault storage backend was added in Vault 0.9.4 (released on February 20, 2018), so ensure you're running Vault 0.9.4 or later before you continue.

Next, create a Cloud Spanner instance and schema for storing our Vault data using the gcloud CLI tool. You can also create the instance and the schema using the web interface or API directly:

$ gcloud spanner instances create my-instance \
  --config=nam3 \
  --description=my-instance \
  --nodes=3

$ gcloud spanner databases create my-database --instance=my-instance

$ gcloud spanner databases ddl update my-database --instance=my-instance --ddl="$(cat <

Next, create a Vault configuration file with the Google Cloud Spanner storage backend configuration: 

# config.hcl
storage "spanner" {
  database   = "projects/my-default-project/instances/my-instance/databases/my-database"
}

Start Vault with the configuration file. This example uses Vault's built-in development mode, which does not represent best practices or a production installation, but it's the fastest way to try the new Cloud Spanner Vault storage backend. 

$ export VAULT_ADDR=http://127.0.0.1:8200
$ sudo vault server -dev -config=config.hcl

During this process, Vault authenticates and connects to Cloud Spanner to populate the data storage layer. After a few seconds, you can view the table data in the web interface and see that data has been populated. Vault is now up-and-running. Again, this is not a production-grade Vault installation. For more details on a production-grade Vault installation, please read the Vault production hardening guide. You can now create, read, update and delete secrets: 

$ vault write secret/my-secret foo=bar

To learn more about the backend configuration options, read the HashiCorp Vault Google Cloud Spanner storage backend documentation. To learn more about Google Cloud Spanner, check out the Google Cloud Spanner documentation.


Toward a great Vault experience on GCP


The Cloud Spanner Vault storage backend enables organizations to leverage the consistency, availability, scalability, replication and security of Cloud Spanner while also supporting Vault's own high availability requirements. In addition to supporting our customers, we're delighted to continue our long-standing relationship with HashiCorp as part of our ongoing partnership. We're excited to see how this new storage backend enables organizations to be more successful with Vault on GCP. Be sure to follow us on Twitter and open a GitHub issue if you have any questions.

Creating a single pane of glass for your multi-cloud Kubernetes workloads with Cloudflare

By Kamilla Amirova, Solutions Engineer, Special Projects, Cloudflare

[Editor’s note: As much as we’d love to host all your workloads on Google Cloud Platform (GCP), sometimes it’s not in the cards. Today we hear from Cloudflare about how to enable a multi-cloud configuration using its load balancer to front Kubernetes-based workloads in both Google Kubernetes Engine and Amazon Web Services (AWS).]

One of the great things about container technology is that it delivers the same experience and functionality across different platforms. This frees you as a developer from having to rewrite or update your application to deploy it on a new cloud provider—or lets you run it across multiple cloud providers. With a containerized application running on multiple clouds, you can avoid lock-in, run your application on the cloud for which it’s best suited, and lower your overall costs.

If you’re using Kubernetes, you probably manage traffic to clusters and services across multiple nodes using internal load-balancing services, which is the most common and practical approach. But if you’re running an application on multiple clouds, it can be hard to distribute traffic intelligently among them. In this blog post, we show you how to use Cloudflare Load Balancer in conjunction with Kubernetes so you can start to achieve the benefits of a multi-cloud configuration.

The load balancers offered by most cloud vendors are often tailored to a particular cloud infrastructure. Load balancers themselves can also be single points of failure. Cloudflare’s Global Anycast Network comprises 120 data centers worldwide and offer all Cloudflare functions, including Load Balancing, to deliver speed and high availability regardless of which clouds your origin servers are hosted on. Users are directed to the closest and most suitable data center to the user, maximizing availability and minimizing latency. Should there be any issue connecting to a given datacenter, user traffic is automatically rerouted to the next best available option. It also health-checks your origins, notifying you via email if one of them is down, while automatic failover capabilities keep your services available to the outside world.

By running containerized applications across multiple clouds, you can be platform-agnostic and resilient to major outages. Cloudflare represents a single pane of glass to:
  • Apply and monitor security policies (DDoS mitigation, WAF, etc.)
  • Manage routing across multiple regions or cloud vendors, using our Load Balancer
  • Tweak performance settings from a single location. This reduces the time you spend managing configurations as well as the possibility of a misconfiguration
  • Add and modify additional web applications as you migrate services from on-premise to cloud or between different cloud providers

Load balancing across AWS and GCP with Cloudflare


To give you a better sense of how to do this, we created a guide on how to deploy an application using Kubernetes on GCP and AWS along with our Cloudflare Load Balancer.

The following diagram shows how the Cloudflare Load Balancer distributes traffic between Google Cloud and another cloud vendor for an application deployed on Kubernetes. In this example, the GCP origin server uses an ingress controller and an HTTP load balancer, while another cloud vendor origin its uses own load balancer. The key takeaway is that Cloudflare Load Balancer works with any of these origin configurations.
Here's an overview of how to set up a load-balanced application across multiple clouds with Cloudflare.

Step 1: Create a container cluster


GCP provides built-in support for running Kubernetes containers with Google Kubernetes Engine. You can access it with Google Cloud Shell, which is preinstalled with gcloud, docker and kubectl command-line tools. Running the following command creates a three-node cluster:

$gcloud container clusters create camilia-cluster --num-nodes=3

Now you have a pool of Compute Engine VM instances running Kubernetes.

AWS


AWS recently announced support for the Kubernetes container orchestration system on top of its Elastic Container Service (ECS). Click Amazon EKS to sign up for the preview.

Until EKS is available, here’s how to create a Kubernetes cluster on AWS:
  • Install the following tools on your local machine: Docker, AWS CLI with an AWS account, Kubectl and Kops (a tool provided by Kubernetes that simplifies the creation of the cluster) 
  • Have a domain name, e.g. mydomain.com
  • In the AWS console have a policy for your user to access the AWS Elastic Container Registry
In addition, you need to have two additional AWS resources in order to create a Kubernetes cluster:
  • An S3 bucket to store information about the created cluster and its configuration 
  • A Route53 domain (hosted zone) on which to run the container, e.g., k8s.mydomain.com. Kops uses DNS for discovery, both inside the cluster and so that you can reach the Kubernetes API server from clients
Once you’ve set up the S3 bucket and created a hosted zone using Kops, you can create the configuration for the cluster and save it on S3: 

$kops create cluster --zones us-east-1a k8saws.usualwebsite.com

Then, run the following command to create the cluster in AWS:

$kops update cluster k8saws.usualwebsite.com --yes

Kops then creates one master node and two slaves. This is the default config for Kops.

Step 2: Deploy the application

This step is the same across both Kubernetes Engine and AWS. After you create a cluster, use kubectl to deploy applications to the cluster. You can usually deploy them from a docker image.

$kubectl run camilia-nginx --image=nginx --port 80

This creates a pod that is scheduled to one of the slave nodes.

Step 3 - Expose your application to the internet 


On AWS, exposing an application to traffic from the internet automatically assigns an external IP address to the service and creates an AWS Elastic Load Balancer.

On GCP, however, containers that run on Kubernetes Engine are not accessible from the internet by default, because they do not have external IP addresses by default. With Kubernetes Engine, you must expose the application as a service internally and create an ingress resource with the ingress controller, which creates an HTTP(S) load balancer.

To expose the application as a service internally, run the following command:

$kubectl expose deployment camilia-nginx --target-port=80  
--type=NodePort

In order to create an ingress resource so that your HTTP(S) web server application is publicly accessible, you'll need to create the yaml configuration file. This file defines an ingress resource that directs traffic to the service.

Once you’ve deployed the ingress resource, the ingress controller that's running in your cluster creates an HTTP(S) Load Balancer to route all external HTTP traffic to the service.

Step 4 - Scale up your application 


Adding additional replicas (pods) is the same for both Kubernetes Engine and AWS. This step ensures there are identical instances running the application.

$kubectl scale deployment camilia-nginx --replicas=3

The Load Balancer that was provisioned in the previous step now starts routing traffic to these new replicas automatically.

Setting up Cloudflare Load Balancer

Now, you’re ready to set up Cloudflare Load Balancer, a very straightforward process:
  • Create a hostname for Load Balancer, for example lb.mydomain.com 
  • Create Origin Pools, for example, a first pool for GCP, and a second pool for AWS 
  • Create Health Checks 
  • Set up Geo Routing, for example all North America East traffic routes to AWS instance, etc.

Please see our documentation for detailed instructions how to set up the Cloudflare Load Balancer.

Introducing Cloudflare Warp


Working with StackPointCloud we also developed a Cloudflare WARP Ingress Controller, which makes it very easy to launch Kubernetes across multiple cloud vendors and uses Cloudflare controller to tie them together. Within StackPointCloud, adding the Cloudflare Warp Ingress Controller requires just a single click. One more click and you've deployed a Kubernetes cluster. Behind the scenes, it implements an ingress controller using a Cloudflare Warp tunnel to connect a Cloudflare-managed URL to a Kubernetes service. The Warp controller manages ingress tunnels in a single namespace of the cluster. Multiple controllers can exist in different namespaces, with different credentials for each namespace.

Kubernetes in a multi-cloud world


With the recent announcement of native Kubernetes support in AWS, as well as existing native support in GCP and Microsoft Azure, it’s clear that Kubernetes is emerging as the leading technology for managing heterogeneous cloud workloads, giving you a consistent way to deploy and manage your applications regardless of which cloud provider they run on. Using Cloudflare Load Balancer in these kinds of multi-cloud configurations lets you direct traffic between clouds, while avoiding vendor-specific integrations and lock-in. To learn more about Cloudflare, visit our website, or reach out to us with any questions — we’d love to hear from you!

Why we used Elastifile Cloud File System on GCP to power drug discovery

By Woody Sherman, Chief Science Officer and Vipin Sachdeva, Principal Investigator, Silicon Therapeutics

[Editor’s note: Last year, Silicon Therapeutics talked about how they used Google Cloud Platform (GCP) to perform massive drug discovery virtual screening. In this guest post, they discuss the performance and management benefits they realized from using the Elastifile Cloud File System and CloudConnect. If you’re looking for a high-performance file system that integrates with GCP, read on to learn more about the environment they built.]

Here, at Silicon Therapeutics, we’ve seen the benefits of GCP as a platform for delivering massive scale-out compute, and have used it as an important component of our drug discovery workload. For example, in our past post we highlighted the use of GCP for screening millions of compounds against a conformational ensemble of a flexible protein target to identify putative drug molecules.

However, like a lot of high-performance computing workflows, we encounter data challenges. It turns out, there are a lot of data management and storage considerations involved with running one of our core applications, molecular dynamics (MD) simulations, which involve the propagation of atoms in a molecular system over time. The time-evolution of atoms is determined by numerically solving Newton's equations of motion, where forces between the atoms are calculated using molecular mechanics force fields. These calculations typically generate thousands of snapshots containing the atomic coordinates, each with tens of thousands of atoms, resulting in relatively large trajectory files. As such, running MD on a large dataset (e.g. the entirety of the ~100,000 structures in the Protein Data Bank (PDB)) could generate a lot of data (over a petabyte).

In scientific computing, decreasing the overall time-to-result and increasing accuracy are crucial in helping to discover treatments for illnesses and diseases. In practice, doing so is extremely difficult due to the ever-increasing volume of data and the need for scalable, high-performance, shared data access and complex workflows. Infrastructure challenges, particularly around file storage, often consume valuable time that could be better spent on core research, thus slowing the progress of critical science.

Our physics-based workflows create parallel processes that generate massive amounts of data, quickly. Supporting these workflows requires flexible, high-performance IT infrastructure. Furthermore, analyzing the simulation results to find patterns and discover new druggable targets means sifting through all that data—in the case of this run, over one petabyte. That kind of infrastructure would be prohibitively expensive to build internally.

The public cloud is a natural fit for our workflows, since in the cloud, we can easily apply thousands of parallel compute nodes to a simulation or analytics job. However, while cloud is synonymous with scalable, high-performance compute, delivering complementary scalable, high-performance storage in the cloud can be problematic. We’re always searching for simpler, more efficient ways to store, manage, and process data at scale, and found that the combination of GCP and the Elastifile cross-cloud data fabric could help us resolve our data challenges, thus accelerating the pace of research.
Our HPC architecture used Google Compute Engine CPUs and GPUs, Elastifile for distributed file storage, and Google Cloud Storage plus Elastifile to manage inactive data.

Why high-performance, scale-out file storage is crucial


To effectively support our bursty molecular simulation and analysis workflows, we needed a cloud storage solution that could satisfy three key requirements:

  • File-native primary storage - Like many scientific computing applications, the analysis software for our molecular simulations was written to generate and ingest data in file format from a file system that ensures strict consistency. These applications won’t be refactored to interface directly with object storage systems like Google Cloud Storage any time soon—hence the need for a cloud-based, POSIX-compliant file system. 
  • Scalable global namespace - Stitching together file servers on discrete cloud instances may suffice for simple analyses on small data sets. However, the do-it-yourself method comes up short as datasets grow and when you need to share data across applications (e.g., in multi-stage workflows). We needed a modern, fully-distributed, shared file system to deliver the scalable, unified namespace that our workflows require. 
  • Cost-effectiveness - Finally, when managing bursty workloads at scale, rigid storage infrastructure can be prohibitively expensive. Instead, we needed a solution that could be rapidly deployed/destroyed, to keep our infrastructure costs aligned to demand. And ideally, for maximum flexibility, we also wanted a solution that could facilitate data portability, both 1) between sites and clouds, and 2) between formats—file format for “active” processing and object format for cost-optimized “inactive” storage/archival/backup.


Solving the file storage problem


To meet our storage needs and support the evolving requirements of our research, we worked with Elastifile, whose cross-cloud data fabric was the backbone of our complex molecular dynamics workflow.

The heart of the solution is the Elastifile Cloud File System (ECFS), a software-only, distributed file system designed for performance and scalability in cloud and hybrid-cloud environments. Built to support the noisy, heterogeneous environments encountered at cloud-scale, ECFS is well-suited to primary storage for data-intensive scientific computing workflows. To facilitate data portability and policy-based controls, Elastifile file systems are exposed to applications via Elastifile “data containers.” Each file system can span any number of cloud instances within a single namespace, while maintaining the strict consistency required to support parallel, transactional applications in complex workflows.

By deploying ECFS on GCP, we were able to simplify and optimize a molecular dynamics workflow. We then applied it to 500 unique proteins as a proof of concept for the aforementioned PDB-wide screen. For this computation, we leveraged a SLURM cluster running on GCP. The compute nodes were 16 n1-highcpu-32 instances, with 8 GPUs attached to every instance for a total of 120 K80 GPUs and 512 CPUs. The storage capacity was provided by a 6 TB Elastifile data container mounted on all the compute nodes.
Defining SLURM configuration to allocate compute and storage resources
Before Elastifile, provisioning and managing storage for such workflows was a complex, manual process. We partitioned the input datasets manually and created several different clusters, each with their own disks. This was because a single large disk often led to NFS issues, specifically with large metadata. In the old world, once the outputs of each cluster were completed, we stored the disks as snapshots. For access, we spun up an instance and shared the credentials for data access. This access pattern was error-prone as well as insecure. Also, at scale, manual processes such as these are time-consuming and introduce risk of critical errors and/or data loss.

With Elastifile, however, deploying and managing storage resources was quick and easy. We simply specified the desired storage capacity, and the ECFS cluster was automatically deployed, configured and made instantly available to the SLURM-managed compute resources . . . all in a matter of minutes. Also, if we want, we can expand the cluster later for additional capacity, with the push of a button. This future-proofs the infrastructure to be able to handle dynamically changing workflow requirements and data scale. By simplifying and automating the deployment process for a cloud-based file system, Elastifile reduced the complexity and risk associated with manual storage provisioning.
Specifying desired file system attributes and policies via Elastifile's unifed management console
In addition, by leveraging Elastifile’s CloudConnect service, we were able to seamlessly promote and demote data between ECFS and Cloud Storage, minimizing infrastructure costs. Elastifile CloudConnect makes it easy to move the data to Google buckets from Elastifile’s data container, and once the data has moved, we can tear down the Elastifile infrastructure, reducing unnecessary costs.
Leveraging Elastifile's CloudConnect UI to monitor progress of data "check in" and "check out" operations between file and object storage
This data movement is essential to our operations, since we need to visualize and analyze subsets of this data on our local desktops. Moving forward, leveraging Elastifile’s combination of data performance, parallelism, scalability, shareability and portability will help us perform more—and larger-scale—molecular analyses in shorter periods of time. This will ultimately help us find better drug candidates, faster.
Visualizing the protein structure, based on the results of the molecular dynamics analyses
As a next step, we’ll work to scale the workflow to all of the unique protein structures in the PDB and perform deep-learning analysis on the resulting data to find patterns associated with proteins dynamics, druggability and tight-binding ligands.

To learn more about how Elastifile supports highly-parallel, on-cloud molecular analysis on GCP, check out this demo video and be sure to visit them at www.elastifile.com.

Solution: Integrating on-premises storage with Google Cloud using an Avere vFXT

By Adrian Graham, Cloud Solutions Architect

Running compute workloads on the cloud can be a powerful way to leverage the massive resources available on Google Cloud Platform (GCP).

Some workloads, such as 3D rendering or HPC simulations, rely on large datasets to complete individual tasks. This isn't a problem when running jobs on-premises, but how do you synchronize tens, or even hundreds of gigabytes of data with cloud storage in order to run the same workload on the cloud?

Even if your Network Attached Storage (NAS) is situated close enough to a Google Cloud datacenter to mount directly, you can quickly saturate your internet connection when hundreds (or even thousands) of virtual machines attempt to read the same data at the same time from your on-premises NAS.

You could implement a synchronization strategy to ensure files exist both on-premises and in the cloud, but managing data concurrency, storage and resources on your own can be challenging, as would be modifying your existing pipeline to perform these actions.

The Avere vFXT is a virtual appliance that provides a solution for such workloads. The vFXT is a cluster of virtual machines that serves as both read-through cache and POSIX-compliant storage. When you mount the vFXT on your cloud instances, your entire on-premises file structure is represented in the cloud. When files are read on the cloud, they're read from your on-premises NAS, across your secure connection, and onto the vFXT. If a file already exists in the vFXT's cache, it's compared with the on-premises version. If the files on both the cache and on-premises are identical, the file is not re-read, which can save you bandwidth and time to first byte.

As cloud instances are deployed, they mount the vFXT as they would any other filesystem (either NFS or SMB). The data is available when they need it, and your connection is spared oversaturation.
We recently helped Avere put together a partner tutorial that shows how to incorporate an Avere vFXT into your GCP project. It also provides guidance on different ways to connect to Google Cloud, and how to access your vFXT more securely and efficiently.

Check out the tutorial, and let us know what other Google Cloud tools you’d like to learn how to use in your visual effects or HPC pipeline. You can reach me on Twitter at @vfx_agraham.

Puppet and Google Cloud Platform: a year in review

By Nelson Araujo, Software Engineer

Here at Google Cloud, our goal is to build a great environment in which to use your favorite management tools, including popular automation and change management software like Puppet. Over the past year, we’ve worked hard to dramatically improve support for Google Cloud in Puppet, and with PuppetConf 2017 behind us, we thought it’d be fun to take a walk down memory lane and recap how far we’ve come.

Our journey to support Puppet-based managed systems started well over a year ago, long before PuppetConf 2016. But that’s where we announced the effort and demoed the proposed model, which described a complete Google Cloud Platform (GCP) infrastructure using Puppet. We collected customer feedback and got busy.

The first wave of integration

Fast forward to this August, when we jointly announced support for the first GCP products on Puppet Forge (Google announcement and Puppet announcement), including:
  • Compute Engine 
  • Kubernetes Engine 
  • Cloud SQL 
  • Cloud DNS
  • Cloud Storage 
  • Unified authentication mechanism
You can find Installation / usage instructions at https://forge.puppet.com/google/cloud and detailed product specific references, usage and examples for the respective module at https://forge.puppet.com/google.

The reception from the community has been great, and these modules have garnered over 1,300 downloads at the time of this writing.

Migrating workloads to Google Cloud Platform


In September, we held a webinar with Puppet to demonstrate how to migrate a Puppet managed application from on-premise (or other cloud provider) to GCP without service interruption.

PuppetConf 2017


The biggest news for GCP and Puppet to come out of PuppetConf 2017 was a formal partnership between the two companies focused on increasing the strength of current and future contributions.

We also announced a second wave of integrations and updates:
  • Cloud Spanner 
  • Cloud Pub/Sub 
  • Stackdriver Logging Agent 
  • Improvements on Compute Engine to support load-balanced / clustered configurations 

You can find Installation / usage instructions at https://forge.puppet.com/google/cloud and detailed product specific references, usage and examples for the respective module at Google @ Puppet Forge.

We also gave a number of talks:
  • From nothing to production in 10 minutes (scalable in 30)
  • Kubernetes in the Cloud with Puppet and Google Kubernetes Engine
And we announced specific new features and integrations:

Bolt / Tasks Support 
We worked with Puppet to add Bolt support to GCP modules, allowing you to run non-idempotent actions that historically did not fit well in Puppet’s model, e.g., restarting a VM. We added a number of tasks to these modules. You can find them in the tasks/ folder of the respective module. We’ve been supporting Bolt since day 1 and will continue doing so in the future.

Puppet Discovery

Puppet unveiled Puppet Discovery during the conference and we're working together to quickly add GCP resource discovery to the Puppet Discovery platform.

What’s next?


We're far from done, and are working on a variety of integration efforts for Puppet, notably:
  • GCP support for Puppet Discovery 
  • Support for Cloud IAM 
  • Support for Cloud Resource Manager 
  • Improvements on Cloud Storage module 
  • Support for Regional Managed Instance Groups (MIG) 
  • Autoscaling of MIGs

If you’re a Puppet user, we hope you give it a try on GCP, and let us know how we’re doing. Reach out to us at [email protected].

Expanding our partner ecosystem with managed services providers

By Zoltan Szabadi, Head of MSP Partners, Google Cloud

When it comes to managing IT for your business, you can never have too much help. That’s why earlier this year we announced that Rackspace was our first managed services provider (MSP) partner for Google Cloud Platform (GCP), and brought its “fanatical support” to customers building on, and migrating to, GCP. Today, the ecosystem of Google Cloud MSPs has continued to expand to a dozen partners globally — and they’re eager to support customers on their journey to the cloud.

For many Google Cloud customers, a qualified MSP makes all the difference. From hands-on support to the ongoing operation of customer workloads, these partners offer proactive services to both large and small cloud adopters. With their staff of dedicated technical experts, MSPs can tackle high-touch projects, covering engagement to migration and execution, to post-planning and ongoing optimization.

Specifically, Google Cloud MSPs offer at minimum:

  • Consulting, assessment, implementation, monitoring and optimization services 
  • 24x7x365 support with enterprise-grade SLAs 
  • L1, L2, L3 tiered support models 
  • Certified support engineers

Our MSP partners can help enterprises on their journey to the cloud — and our MSP partners can help. All are highly knowledgeable about Google Cloud technologies and processes, are investing dedicated engineering resources to support GCP customers and participate in immersive Google training that includes role and team shadowing.

Meet the Google Cloud MSPs


Here are the partners that now make up our MSP ecosystem:

  • Accenture - Accenture is a global leader in helping organizations move to cloud. Their managed service capability is multi-mode and multi-cloud, and they work with GCP to help customers with security, data, application, infrastructure or business process operations.
  • Cascadeo - As a long-time Google partner, Cascadeo specializes in managed services, cloud operations and automation. Their services are designed to allow customers to bring their own tooling, monitoring systems, and accounts — while leveraging best in breed services and engineers.
  • Claranet - With more than 1,800 employees and 24 offices across Europe and Brazil, Claranet specializes in helping customers migrate non-'cloud native' workloads into GCP, and then iteratively improve the infrastructure to take advantage of all of GCP’s tools and services.
  • Cloudreach -  Cloudreach has transitioned some of the largest enterprises from traditional IT to the power of GCP. They've been named a Gartner Magic Quadrant Leader for Hyperscale Cloud Managed Services and offer software enabled services including migration, app modernization, financial, security and governance management.
  • DoiT International - DoIT’s team of GCP experts provides state-of-the-art managed services for startups. They focus on helping some of the most innovative startups in the world design best in class architecture, cloud operations, security protection and advanced big data analytics.
  • Go Reply - Go Reply, the Reply group company specialized on Google Cloud technology, offers design and implementation on GCP. They help enterprises build and optimize their cloud strategy, migration, infrastructure hosting, big data, machine learning, PCI/ISO compliance and security management.
  • Pythian - For over 20 years, Pythian has been delivering managed services. They feature flexible subscription packages, with mix and match services, that help control costs while maximizing performance, availability, efficiency and security — ensuring customers get the most from their GCP environments.
  • Rackspace - Rackspace helps customers of all sizes plan, architect and operate projects at scale leveraging its 150+ GCP-certified experts. Rackspace recently acquired Datapipe, a next-gen cloud MSP, and both were named as leaders in the 2017 Gartner Magic Quadrant for Public Cloud Infrastructure Managed Service Providers, Worldwide.
  • RightScale - RightScale provides managed services to help organizations successfully and cost-effectively deploy and manage applications. Through the RightScale Cloud Management Platform and RightScale Optima for cost management, they enable enterprises, other MSPs, and system integrators to grow their cloud business on GCP.
  • SADA Systems - SADA Systems works with GCP customers at various stages of their cloud adoption strategy. They assist with deploying new cloud-native applications, augmenting existing applications to run more efficiently in the cloud or the hybrid cloud — and offer security services, billing management and automation via Orbitera.
  • Sutherland - As a unique Support Partner for Google Cloud, Sutherland brings domain expertise, process transformation, design knowledge and combined machine learning and artificial intelligence application management to GCP customers.
  • Taos - Taos delivers a robust portfolio of cloud, on-premise and hybrid managed services, assessments and migrations — plus end-to-end customer support for the full GCP journey. They leverage GCP to create future-proof infrastructure, data and analytics to mitigate concerns about reliability, capacity or performance.

We’ll be announcing new MSP partners in the coming months, so keep an eye on our partner page. You can also learn more about our partner program, find a partner or read some of the cool customer stories that our partners have made possible. We’re excited to see what you’ll build together!

Announcing integration of Altair HPC applications with Google Cloud

By Michael Basilyan, Product Manager and Annie Ma-Weaver, Partner Manager 

Engineering today requires access to unprecedented computing resources to simulate, test and design the products that make modern life possible. Here at Google Cloud, one of our goals is to democratize and simplify access to advanced computing resources and promote the sciences and engineering.

With that, we’re excited to announce a new technology partnership between Google Cloud Platform (GCP), Intel and Altair, a leading software provider for engineering and science applications, including high performance computing (HPC) applications for computer-aided engineering, simulation, product design, Internet of Things and others.

Starting today, you can launch virtual HPC appliances running Altair and other HPC applications on GCP using Altair’s PBScloud.io. PBScloud provides a central command center, a simple user experience, easy deployment, real-time monitoring and resource management for HPC use cases. It also includes features for job submission, job monitoring and result visualization. PBScloud.io also works and orchestrates across multiple public clouds and traditional on-premises deployments.
Altair applications available on GCP via PBScloud.io
Before cloud computing, engineers and scientists were constrained by the limitations of on-premise computing resources and clusters. Long queue times, suboptimal hardware utilization and frustrated users were commonplace. With Google Cloud, you can test your ideas quickly, pay for exactly what you need and only while you need it. Now with Altair’s PBScloud.io, you also have easy, turn-key access to state-of-the-art science and engineering applications on GCP’s advanced, scalable hardware and infrastructure.

Compare, for example, the performance of Altair RADIOSS on Intel’s latest generation Xeon processor codenamed Skylake on Compute Engine vs. its performance on previous generation CPUs. Note that RADIOSS demonstrated product scalability by taking advantage of all 96 vCPUs on GCP.


We’re excited to bring this collaboration to you and even more excited to see what you'll build with Altair’s software on our platform. If you’re at the SC17 conference, be sure to drop by the Google Cloud, Altair and Intel booths for talks, demos and to talk about HPC on Google Cloud.

Check out PBScloud.io and sign up for a GCP trial at no cost today.

Commvault and Google Cloud partner on cloud-based data protection and simpler “lift and shift” to the cloud

By Adam Massey, Director, Strategic Technology Partnerships

Today at Commvault Go 2017, we announced a new strategic alliance with Commvault to enable you to benefit from advanced data protection in the cloud as well as on-premises, and to make it easier to “lift-and-shift” workloads to Google Cloud Platform (GCP).

At Google Cloud, we strive to provide you with the best offerings not just to store but also to use your data. For example, if you’re looking for data protection, you can benefit from our unique Coldline class as part of Google Cloud Storage, which provides immediate access to your data at archival storage prices. You can test this for free. Try serving an image or video directly from the Coldline storage tier and it will return within milliseconds. Then there’s our partner Forsythe, whose data analytics-as-a-service offering allows you to bring your backup data from Commvault to Google Cloud Storage and then analyze it using GCP machine learning and data loss prevention services.

We work hard with our technology partners to deliver solutions that are easy to use and cost-effective. We're working with Commvault on a number of initiatives, specifically:
  • Backup to Google Cloud Storage Coldline: If you use Commvault, you can now use Coldline in addition to Regional and Nearline classes as your storage target. Check out this video to see how easy it is to set up Cloud Storage with Commvault.
  • Protect workloads in the cloud: As enterprises move their applications to Google Compute Engine, you can use the same data protection policies that you use on-premises with Commvault’s data protection software. Commvault supports a wide range of common enterprise applications from SAP, Exchange, SQL, DB2, and PostgreSQL, to big data applications such as GPFS, MongoDB, Hadoop and many more.
  • G Suite backup with Commvault: You can now use the Commvault platform to backup and recover data from G Suite applications such as Gmail and Drive.
We're excited to work with Commvault to bring more capabilities to our joint customers in the future, such as enhanced data visibility via analytics and the ability to migrate and/or recover VMs in Compute Engine for on-premises workloads.

If you’re planning to attend Commvault Go this week, visit our booth to learn more about our partnership with Commvault and how to use GCP for backup and disaster recovery with Commvault!

Introducing Grafeas: An open-source API to audit and govern your software supply chain

By Stephen Elliott, Product Manager, Developer Platforms and  Jianing Guo, Product Manager, Container Security 

Building software at scale requires strong governance of the software supply chain, and strong governance requires good data. Today, Google, along with JFrog, Red Hat, IBM, Black Duck, Twistlock, Aqua Security and CoreOS, is pleased to announce Grafeas, an open source initiative to define a uniform way for auditing and governing the modern software supply chain. Grafeas (“scribe” in Greek) provides organizations with a central source of truth for tracking and enforcing policies across an ever growing set of software development teams and pipelines. Build, auditing and compliance tools can use the Grafeas API to store, query and retrieve comprehensive metadata on software components of all kinds.

As part of Grafeas, Google is also introducing Kritis, a Kubernetes policy engine that helps customers enforce more secure software supply chain policies. Kritis (“judge” in Greek) enables organizations to do real-time enforcement of container properties at deploy time for Kubernetes clusters based on attestations of container image properties (e.g., build provenance and test status) stored in Grafeas.
“Shopify was looking for a comprehensive way to track and govern all the containers we ship to production. We ship over 6,000 builds every weekday and maintain a registry with over 330,000 container images. By integrating Grafeas and Kritis into our Kubernetes pipeline, we are now able to automatically store vulnerability and build information about every container image that we create and strictly enforce a built-by-Shopify policy: our Kubernetes clusters only run images signed by our builder. Grafeas and Kritis actually help us achieve better security while letting developers focus on their code. We look forward to more companies integrating with the Grafeas and Kritis projects.”  
— Jonathan Pulsifer, Senior Security Engineer at Shopify. (Read more in Shopify’s blog post.)

The challenge of governance at scale 


Securing the modern software supply chain is a daunting task for organizations both large and small, exacerbated by several trends:

  • Growing, fragmented toolsets: As an organization grows in size and scope, it tends to use more development languages and tools, making it difficult to maintain visibility and control of its development lifecycle. 
  • Open-source software adoption: While open-source software makes developers more productive, it also complicates auditing and governance. 
  • Decentralization and continuous delivery: The move to decentralize engineering and ship software continuously (e.g., “push on green”) accelerates development velocity, but makes it difficult to follow best practices and standards. 
  • Hybrid cloud deployments: Enterprises increasingly use a mix of on-premises, private and public cloud clusters to get the best of each world, but find it hard to maintain 360-degree visibility into operations across such diverse environments. 
  • Microservice architectures: As organizations break down large systems into container-based microservices, it becomes harder to track all the pieces.

As a result, organizations generate vast quantities of metadata, all in different formats from different vendors and are stored in many different places. Without uniform metadata schemas or a central source of truth, CIOs struggle to govern their software supply chains, let alone answer foundational questions like: “Is software component X deployed right now?” “Did all components deployed to production pass required compliance tests?” and “Does vulnerability Y affect any production code?” 

The Grafeas approach 

Grafeas offers a central, structured knowledge-base of the critical metadata organizations need to successfully manage their software supply chains. It reflects best practices Google has learned building internal security and governance solutions across millions of releases and billions of containers. These include:

  • Using immutable infrastructure (e.g., containers) to establish preventative security postures against persistent advanced threats 
  • Building security controls into the software supply chain, based on comprehensive component metadata and security attestations, to protect production deployments 
  • Keeping the system flexible and ensuring interoperability of developer tools around common specifications and open-source software

Grafeas is designed from the ground up to help organizations apply these best practices in modern software development environments, using the following features and design points:

  • Universal coverage: Grafeas stores structured metadata against the software component’s unique identifier (e.g., container image digest), so you don’t have to co-locate it with the component’s registry, and so it can store metadata about components from many different repositories. 
  • Hybrid cloud-friendly: Just as you can use JFrog Artifactory as the central, universal component repository across hybrid cloud deployments, you can use the Grafeas API as a central, universal metadata store. 
  • Pluggable: Grafeas makes it easy to add new metadata producers and consumers (for example, if you decide to add or change security scanners, add new build systems, etc.) 
  • Structured: Structured metadata schemas for common metadata types (e.g., vulnerability, build, attestation and package index metadata) let you add new metadata types and providers, and the tools that depend on Grafeas can immediately understand those new sources. 
  • Strong access controls: Grafeas allows you to carefully control access for multiple metadata producers and consumers. 
  • Rich query-ability: With Grafeas, you can easily query all metadata across all of your components so you don’t have to parse monolithic reports on each component.

Defragmenting and centralizing metadata 

At each stage of the software supply chain (code, build, test, deploy and operate), different tools generate metadata about various software components. Examples include the identity of the developer, when the code was checked in and built, what vulnerabilities were detected, what tests were passed or failed, and so on. This metadata is then captured by Grafeas. See the image below for a use case of how Grafeas can provide visibility for software development, test and operations teams as well as CIOs.
(click to enlarge)

To give a comprehensive, unified view of this metadata, we built Grafeas to promote cross-vendor collaboration and compatibility; we’ve released it as open source, and are working with contributors from across the ecosystem to further develop the platform:

  • JFrog is implementing Grafeas in the JFrog Xray API and will support hybrid cloud workflows that require metadata in one environment (e.g., on-premises in Xray) to be used elsewhere (e.g., on Google Cloud Platform). Read more on JFrog’s blog. 
  • Red Hat is planning on enhancing the security features and automation of Red Hat Enterprise Linux container technologies in OpenShift with Grafeas. Read more on Red Hat’s blog. 
  • IBM plans to deliver Grafeas and Kristis as part of the IBM Container Service on IBM Cloud, and to integrate our Vulnerability Advisor and DevOps tools with the Grafeas API. Read more on IBM’s blog. 
  • Black Duck is collaborating with Google to implement the Google artifact metadata API implementation of Grafeas, to bring improved enterprise-grade open-source security to Google Container Registry and Google Container Engine. Read more on Black Duck’s blog. 
  • Twistlock will integrate with Grafeas to publish detailed vulnerability and compliance data directly into orchestration tooling, giving customers more insight and confidence about their container operations. Read more on Twistlock’s blog.
  • Aqua Security will integrate with Grafeas to publish vulnerabilities and violations, and to enforce runtime security policies based on component metadata information. Read more on Aqua’s blog. 
  • CoreOS is exploring integrations between Grafeas and Tectonic, its enterprise Kubernetes platform, allowing it to extend its image security scanning and application lifecycle governance capabilities. 

Already, several contributors are planning upcoming Grafeas releases and integrations:

  • JFrog’s Xray implementation of Grafeas API 
  • A Google artifact metadata API implementation of Grafeas, together with Google Container Registry vulnerability scanning 
  • Bi-directional metadata sync between JFrog Xray and the Google artifact metadata API 
  • Black Duck integration with Grafeas and the Google artifact metadata API 
Building on this momentum, we expect numerous other contributions to the Grafeas project early in 2018.

Join us!

The way we build and deploy software is undergoing fundamental changes. If scaled organizations are to reap the benefits of containers, microservices, open source and hybrid cloud, they need a strong governance layer to underpin their software development processes. Here are some ways you can learn more about and contribute to the project:


 We hope you will join us!

Partnering on open source: Managing Google Cloud Platform with Chef

By Nelson Araujo, Software Engineer

Managing cloud resources is a critical part of the application lifecycle. That’s why today, we released and open sourced a set of comprehensive cookbooks for Chef users to manage Google Cloud Platform (GCP) resources.

Chef is a continuous automation platform powered by an awesome community. Together, Chef and GCP enable you to drive continuous automation across infrastructure, compliance and applications.

The new cookbooks allow you to define an entire GCP infrastructure using Chef recipes. The Chef server then creates the infrastructure, enforces it, and ensures it stays in compliance. The cookbooks are idempotent, meaning you can reapply them when changes are required and still achieve the same result.

The new cookbooks support the following products:



We also released a unified authentication cookbook that provides a single authentication mechanism for all the cookbooks.

These new cookbooks are Chef certified, having passed the Chef engineering team’s rigorous quality and review bar, and are open-source under the Apache-2.0 license on GCP's Github repository.

We tested the cookbooks on CentOS, Debian, Ubuntu, Windows and other operating systems. Refer to the operating system support matrix for compatibility details. The cookbooks work with Chef Client, Chef Server, Chef Solo, Chef Zero, and Chef Automate.

To learn more about these Chef cookbooks, register for the webinar with myself and Chef’s JJ Asghar on 15 October 2017.

Getting started with Chef on GCP

Using these new cookbooks is as easy as following these four steps:
  1. Install the cookbooks.
  2. Get a service account with privileges for the GCP resources that you want to manage and enable the the APIs for each of the GCP services you will use.
  3. Describe your GCP infrastructure in Chef:
    1. Define a gauth_credential resource
    2. Define your GCP infrastructure
  4. Run Chef to apply the recipe.
Now, let’s discuss these steps in more detail.

1. Install the cookbooks

You can find all the GCP cookbooks for Chef on Chef Supermarket. We also provide a “bundle” cookbook that installs every GCP cookbook at once. That way you can choose the granularity of the code you pull into your infrastructure.

Note: These Google cookbooks require neither administrator privileges nor special privileges/scopes on the machines that Chef runs on. You can install the cookbooks either as a regular user on the machine that will execute the recipe, or on your Chef server; the latter option distributes the cookbooks to all clients.

The authentication cookbook requires a few of our gems. You can install them using various methods, including using Chef itself:


chef_gem 'googleauth'
chef_gem 'google-api-client'


For more details on how to install the gems, please visit the authentication cookbook documentation.

Now, you can go ahead and install the Chef cookbooks. Here’s how to install them all with a single command:


knife cookbook site install google-cloud


Or, you can install only the cookbooks for select products:


knife cookbook site install google-gcompute    # Google Compute Engine
knife cookbook site install google-gcontainer  # Google Container Engine
knife cookbook site install google-gdns        # Google Cloud DNS
knife cookbook site install google-gsql        # Google Cloud SQL
knife cookbook site install google-gstorage    # Google Cloud Storage


2. Get your service account credentials and enable APIs

To ensure maximum flexibility and portability, you must authenticate and authorize GCP resources using service account credentials. Using service accounts allows you to restrict the privileges to the minimum necessary to perform the job.

Note: Because service accounts are portable, you don’t need to run Chef inside GCP. Our cookbooks run on any computer with internet access, including other cloud providers. You might, for example, execute deployments from within a CI/CD system pipeline such as Travis or Jenkins, or from your own development machine.

Click here to learn more about service accounts, and how to create and enable them.

Also make sure to enable the the APIs for each of the GCP services you intend to use.

3a. Define your authentication mechanism

Once you have your service account, add the following resource block to your recipe to begin authenticating with it. The resource name, here 'mycred' is referenced in the objects in the credential parameter.


gauth_credential 'mycred' do
  action :serviceaccount
  path '/home/nelsonjr/my_account.json'
  scopes ['https://www.googleapis.com/auth/compute']
end


For further details on how to setup or customize authentication visit the Google Authentication cookbook documentation.

3b. Define your resources

You can manage any resource for which we provide a type. The example below creates an SQL instance and database in Cloud SQL. For the full list of resources that you can manage, please refer to the respective cookbook documentation link or to this aggregate summary view.


gsql_instance ‘my-app-sql-server’ do
  action :create
  project 'google.com:graphite-playground'
  credential 'mycred'
end

gsql_database 'webstore' do
  action :create
  charset 'utf8'
  instance ‘my-app-sql-server’
  project 'google.com:graphite-playground'
  credential 'mycred'
end


Note that the above code has to be described in a recipe within a cookbook. We recommend you have a “profile” wrapper cookbook that describes your infrastructure, and reference the Google cookbooks as a dependency.

4. Apply your recipe

Next, we direct Chef to enforce the recipe in the “profile” cookbook. For example:

$ chef-client -z --runlist ‘recipe[mycloud::myapp]’

In this example, mycloud is the “profile” cookbook, and myapp is the recipe that contains the GCP resource declarations.

Please note that you can apply the recipe from anywhere that Chef can execute recipes (client, server, automation), once or multiple times, or periodically in the background using an agent.

Next steps

Now you're ready to start managing GCP resources with Chef, and start reaping the benefits of cross-cloud configuration management. Our plan is to continue to improve the cookbooks and add support for more Google products. We're also preparing to release the technology used to create these cookbooks as open source. If you have questions about this effort please visit Chef on GCP discussions forum, or reach out to us on [email protected].