Posted by Vic Iglesias, Cloud Solutions Architect

Zookeeper is the cornerstone of many distributed systems, maintaining configuration information, naming and providing distributed synchronization and group services. But using Zookeeper in a dynamic cloud environment can be challenging, because of the way it keeps track of members in the cluster. Luckily, there’s an open source package called Exhibitor that makes it simple to use Zookeeper with Google Container Engine.

With Zookeeper, it’s easy to implement the primitives for coordinating hosts, such as locking and leader election. To provide these features in a highly available manner, Zookeeper uses a clustering mechanism that requires a majority of the cluster members to acknowledge a change before the cluster’s state is committed. In a cloud environment, however, Zookeeper machines come and go from the cluster (or “ensemble” in Zookeeper terms), changing names and IP addresses, and losing state about the ensemble. These sorts of ephemeral cloud instances are not well suited to Zookeeper, which requires all hosts to know the addresses or hostnames of the other hosts in the ensemble.


Netflix tackled this issue, and in order to more easily configure, operate, and debug Zookeeper in cloud environments, created Exhibitor, a supervisor process that coordinates the configuration and execution of Zookeeper processes across many hosts. Exhibitor also provides the following features for Zookeeper operators and users:

• Backup and restore
• A lightweight GUI for Zookeeper nodes
• A REST API for getting the state of the Zookeeper ensemble
• Rolling updates of configuration changes

Let’s take a look at using Exhibitor in Container Engine in order to run Zookeeper as a shared service for our other applications running in a Kubernetes cluster.

To start, provision a shared file server to host the shared configuration file between your cluster hosts. The easiest way to get a file server up and running in Google Cloud Platform is to create a Single Node File Server using Google Cloud Launcher. For horizontally scalable and highly available shared filesystem options on Cloud Platform, have a look at Red Hat Gluster Storage and Avere. The resulting file server will expose both NFS and SMB file shares.

1. Provision your file server
2. Select the Cloud Platform project in which you’d like to launch it
3. Choose the following parameters for the dimensions of your file server
• Zone — must match the zone of your Container Engine cluster (to be provisioned later)
• Machine type — for this tutorial we chose a n1-standard-1 as throughput will not be very high for our hosted files
• Storage disk size — for this tutorial, we chose a 100GB disk as we will not need high throughput nor IOPS
4. Click Deploy

Next, create a Container Engine cluster in which to deploy your Zookeeper ensemble.

1. Create a new Kubernetes cluster using Google Container Engine
2. Ensure that the zone setting is the same as you use to deploy your file server (for the purposes of this tutorial, leave the defaults for the other settings)
3. Click Create

Once your cluster is created, open up the Cloud Shell in the Cloud Console by

1. Set the default zone for the gcloud CLI to the zone you created for your file server and Kubernetes cluster:

gcloud config set compute/zone us-east1-d

2. Download your Kubernetes credentials via the Cloud SDK:

gcloud container clusters get-credentials cluster-1

3. Create a file named exhibitor.yaml with the following content. This will define our Exhibitor deployment as well as the service that other applications can use to communicate with it:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: exhibitor
spec:
 replicas: 3
 template:
   metadata:
     labels:
       name: exhibitor
   spec:
    containers:
    - image: mbabineau/zookeeper-exhibitor
    imagePullPolicy: Always
    name: exhibitor
    volumeMounts:
      - name: nfs
      mountPath: "/opt/zookeeper/local_configs"
    livenessProbe:
      tcpSocket:
        port: 8181
      initialDelaySeconds: 60
      timeoutSeconds: 1
    readinessProbe:
      httpGet:
        path: /exhibitor/v1/cluster/4ltr/ruok
        port: 8181
      initialDelaySeconds: 120
      timeoutSeconds: 1
     env:
     - name: HOSTNAME
       valueFrom:
         fieldRef:
           fieldPath: status.podIP
    volumes:
    - name: nfs
      nfs:
        server: singlefs-1-vm
        path: /data
---
apiVersion: v1
kind: Service
metadata:
  name: exhibitor
  labels:
    name: exhibitor
spec:
  ports:
  - port: 2181
    protocol: TCP
    name: zk
  - port: 8181
    protocol: TCP
    name: api
  selector:
    name: exhibitor

In this manifest we’re configuring the NFS volume to be attached on each of the pods and mounted in the folder where Exhibitor expects to find its shared configuration.
4. Create the artifacts in that manifest with the kubectl CLI

kubectl apply -f exhibitor.yaml

5. Monitor the pods until they all enter the RUNNING state

kubectl get pods -w

6. Run the kubectl proxy command so that you can access the Exhibitor REST API for the Zookeeper state:

kubectl proxy &

7. Query the Exhibitor API using curl, then use jq to format the JSON response to be more human-readable

export PROXY_URL=http://localhost:8001/api/v1/proxy/
export EXHIBITOR_URL=${PROXY_URL}namespaces/default/services/exhibitor:8181
export STATUS_URL=${EXHIBITOR_URL}/exhibitor/v1/cluster/status
export STATUS=`curl -s $STATUS_URL`
echo $STATUS | jq '.[] | {hostname: .hostname, leader: .isLeader}'

8. After a few minutes, the cluster state will have settled and elected a stable leader.

You have now completed the tutorial and can use your Exhibitor/Zookeeper cluster just like any other Kubernetes service, by accessing its exposed ports (2181 for Zookeeper and 8181 for Exhibitor) and addressing it via its DNS name, or by using environment variables.

In order to access the Exhibitor web UI, run the kubectl proxy command on a machine with a browser, then visit this page.

To scale up your cluster size, simply edit the exhibitor.yaml file and change the replicas to an odd number greater than 3 (e.g., 5), then run “kubectl apply -f exhibitor.yaml” again. Cause havoc in the cluster by killing pods and nodes in order to see how it responds to failures.

Posted by Luke Stone, Director of Technical Support

In December 2011, I had been working for Google for nine years and was leading a team of 10 software developers, supporting the AdSense business. Our portfolio consisted of over 30 software systems, mostly web apps for business intelligence that had been built over the past decade, each on a stack that seemed like a good idea at the time. Some were state-of-the-art custom servers built on the (then) latest Google web server libraries and running directly on Borg. Some were a LAMP stack on a managed hosting service. Some were running as a cron job on someone’s workstation. Some were weird monsters, like a LAMP stack running on Borg with Apache customized to work with production load balancers and encryption. Things were breaking in new and wonderful ways every day. It was all we could do to keep the systems running — just barely.

The team was stressed out. The Product Managers and engineers were frustrated. A typical conversation went like this:

          PM: “You thought it would be easy to add the foobar feature, but it’s been four
          days!”
          Eng: “I know, I know, but I had to upgrade the package manager version
          first, and then migrate off some deprecated APIs. I’m almost done with that stuff.
          I’m eager to start on the foobar,  too.”
          PM: “Well, now, that's disappointing.”

I surveyed the team to find the root cause of our inefficiency: we were spending 60% of our time on maintenance. I asked how much time would be appropriate, and the answer was a grudging 25%. We made a goal to reduce our maintenance to that point, which would free up the time equivalent of three and a half of our 10 developers.

Google App Engine had just come out of preview in September 2011. A friend recommended it heartily — he'd been using it for a personal site — and he raved that it was low-maintenance, auto-scaling and had built-in features like Google Cloud Datastore and user-management. Another friend, Alex Martelli, was using it for several personal projects. I myself had used it for a charity website since 2010. We decided to use it for all of our web serving. It was the team’s first step into PaaS.

Around the same time, we started using Dremel, Google’s internal version of BigQuery. It was incredibly fast compared to MapReduce, and it scaled almost as well. We decided to re-write all of our data processing to use it, even though there were still a few functional gaps between it and App Engine at the time, for example visualization and data pipelines. We whipped up solutions that are still in use by hundreds of projects at Google. Now Google Cloud Platform users can access similar functionality using Google Cloud Datalab.

What we saw next was an amazing transformation in the way that software developers worked. Yes, we had to re-write 30 systems, but they needed to be re-written anyway. WIth that finished, developing on the cloud was so much faster -- I recall being astonished at seeing the App Engine logs, that I had done 100 code, test, and deploy cycles in a single coding session. Once things were working, they kept working for a long time. We stopped debating what stack to choose for the next project. We just grabbed the most obvious one from Google Cloud Platform and started building. If we found a bug in the cloud infrastructure, it was promptly fixed by an expert. What a change from spending hours troubleshooting library compatibility!

Best of all, we quickly got the time we spent on maintenance down to 25%, and it kept going down. At the end of two years I repeated the survey; the team reported that they now only spent 5% of their time on maintenance.

We started having good and different problems. The business wasn’t generating ideas fast enough to keep us busy, and we had no backlog. We started to take two weeks at the end of every quarter for a “hackathon” to see what we could dream up. We transferred half of the developers to another, busier team outside of Cloud. We tackled larger projects and started out-pacing much larger development teams.

After seeing how using PaaS changed things for my team, I want everyone to experience it. Thankfully, these technologies are available not only to Google engineers, but to developers the world over. This is the most transformational technology I’ve seen since I first visited Google Search in 1999 — it lets developers stop doing dumb things and get on with developing the applications that add value to our lives.

Posted by Scott Van Woudenberg, Product Manager, Google Cloud Platform


It's been a little over a year (and millions of migrations) since the last time we talked about our live migration technology and how we use it to keep your virtual machines humming along while we patch, repair and update the software and hardware infrastructure that powers Google Compute Engine. It’s also an important differentiator for our platform compared to other cloud providers.

Our customer base has grown exponentially in the past year, and brought with it a lot of new and interesting workloads to test the mettle of our live migration technology. The vast majority of customers and workloads have been able to go about their business without noticing our maintenance events, with a few exceptions.

Down and to the right

A picture is worth 1,000 words and the following graph shows the improvements we've made to live migration blackout times (the amount of time your VM is paused) over the last year (note the log scale):



We've done millions of live migrations in the last year and as you can see from the graph, we've made significant improvements to median blackout duration and variance. The 99th percentile blackout graph is too noisy to display nicely, but we've improved that by a factor of six in the last year as well.

Lessons learned

The graph also shows that we didn't always get it right, and we've learned a lot from working closely with the handful of customers whose applications just don't play well with live migration.

The most important thing we learned is that the current 60 second pre-migration signal is overkill for the vast majority of customers. At the same time, the 60 second signal is too short for the handful of customers that needed to perform some sort of automated drain or failover action.

We also learned that the older in-place upgrade maintenance mechanism (not captured in the above graph) we use to update our hypervisor is problematic for customers whose applications are sensitive to live migration blackouts.

Finally, we learned that surfacing VM migrations as system events in our ZoneOperations list led to a lot of confusion for little incremental value, since the events were also logged, in greater detail, in Compute Engine’s Cloud Logs. In many cases, customers noticed an issue with their service, saw the migration system event in the ZoneOperations list, and spent a long time investigating that as the cause, only to find that it was a red herring.

What's next?

Aside from continuing to measure and improve the impact of our maintenance events, the first thing we're going to do is drop the 60 second notice for VMs that don't care about it. If a VM is actively monitoring the metadata maintenance event URI, we'll continue to give 60 seconds notice before the migration. If a VM is not monitoring the URI, however, we'll start the migration immediately. This will not change the behavior of VMs that are configured to terminate instead of migrate. We expect to roll this out by mid-May.

In the coming quarters we’ll also begin providing a longer, more usable advance notice for VMs that are configured to terminate instead of migrate. This advance notice signal will be available via the metadata server as well as via the Compute Engine API on the Instance resource.

We'll also add a new API method to the Instance resource to allow customers to trigger maintenance events on a VM. This will give customers a means to determine if a VM is impacted by maintenance events and if so, to validate that its drain/failover automation works correctly.

The second major change we'll make is to use live migration for all instance virtualization software stack updates, replacing in-place upgrades. This will make these maintenance events visible, actionable and less disruptive while allowing us to focus our improvements on one maintenance mechanism rather than two. Finally, we'll remove VM migration events from our Operations collections and expand on the details of the events we log in Cloud Logs.

We strongly believe that fast, frequent, reliable, and above all, transparent infrastructure maintenance is essential to keeping our systems secure and to delivering the new features and services that customers want. We're pleased with the results we've seen so far, and we're excited to continue making it better. Follow us on G+ and/or Twitter to stay informed as we start rolling out these improvements. If you have any questions or concerns, please reach out — we'd love to hear from you.

Posted by Alex Barrett, Content Writer & Editor, Google Cloud Platform

Couldn’t make it to GCP NEXT last month? Lucky for you, recordings of over thirty sessions are up on YouTube to watch and learn from.

But with so much quality content to watch, where do you start? Leaving aside the wildly popular Data Center 360 video that tours the Dalles, Oregon facility (one million views and counting), and must-see Day One keynotes and Day 2 keynotes, here are the five most popular GCP NEXT breakout sessions on YouTube.

1. What’s the secret behind Google’s legendary uptime? Three words: site reliability engineers. In “How Google Does Planet-Scale Engineering for Planet-Scale Infrastructure” Google engineering director Melissa Bindle lays out the relationship between SREs and developer teams, and how they ensure uptime for Google.com and Google Cloud Platform users. Even if your organization isn’t operating at quite planetary scale, lessons abound for anyone that cares about building and operating resilient environments.
2. What exactly is machine learning, and what can it do for you? In “Build smart applications with your new superpower: cloud machine learning” Google Developer Advocate Julia Ferraioli walks through the range of Google’s machine learning options, from TensorFlow, to Google Cloud Machine Learning to the Machine Learning APIs. She’s also joined by David Zuckerman, head of developer experience at Wix.com, who describes how it used Google machine learning to mine customer images with the new Vision API.
3. Cloud Platform users do the darndest things. In Analyzing 25 billion stock market events in an hour with NoOps on GCP, Fidelity Information Systems CTO Neil Palmer and developer Todd Ricker describe how they “ingest, process and analyze the entire U.S. options and equities market in 4 hours using Google Cloud Dataflow, Google Coud Pub/Sub and Google Cloud Bigtable.” That’s 15 TB of data. On a daily basis. That it then makes available for market reanalysis to the regulators.
4. Cloud Platform isn’t all about crunching large amounts of data for ginormous financial services firms. It’s also about democratizing access to Google’s amazing capabilities to the world at large, wherever they may be. In “Building iOS Apps with Firebase,” Google Cloud Developer Advocate Sara Robinson walks users through how to develop a native iOS app using Firebase, a mobile back-end as a service, and then ties that app into Google Cloud Vision API to provide face and object recognition.
5. Rounding out the list of most-watched breakout sessions is Painless container management with GKE & Kubernetes. Brendan Burns, Google’s lead engineer on the Kubernetes project, provides context and backstory; Rich Steenberg, principal engineer at WePay, a Kubernetes shop, describes how they’re using Kubernetes today; and Tim Hockin, a Google software engineer, describes what’s new in Kubernetes 1.2, and what’s on tap.

Did your favorite session not make the list? Share it in the comments so the rest of the world can benefit from what you learned.

Posted by Ben Chong, Product Manager


Iron Mountain has expanded the bandwidth of its network link to Google Cloud Platform by 10x. This enables faster transmission of your data to GCP, which is super helpful for meeting critical project deadlines. As an illustration of transmission speed, moving 50TB of data over the expanded link takes less than a day, compared to more than five days with the old connection.

Iron Mountain has been providing cloud seeding services for our customers in North America since 2015.

Enterprises are eager to run applications on public clouds to benefit from the security, scalability and reduced management burden. However, moving data to the public cloud as part of a migration or hybrid process is slow and troublesome. Uploading one terabyte of data using a typical small business DSL connection may take up to 100 days!

As a result, enterprises are frequently turning to cloud seeding services, also referred to as offline ingestion or offline data import, from providers like Iron Mountain.

Instead of trying to push data to a public cloud over a limited bandwidth, customers copy data to storage devices and ship them to a third party who can then securely upload the data to the customers’ data buckets in cloud storage. Cloud seeding providers include, as part of their service portfolio, chain-of-custody verification and security options like encryption to help ensure that customer data is handled securely.

Iron Mountain has also introduced support for LTO (Linear Tape-Open) tapes, which is a good transport medium for large amounts of data. Customers with data already on LTO media can immediately start moving that data into low cost Google Cloud Storage Nearline.

You could be eligible for up to $500 of credit on Cloud Platform if you ingest with Iron Mountain! More information can be found here. If you’re ready to start a project on Cloud Platform and are not sure how to start moving your data to the cloud, we’ve got you covered with offline data import.

Posted by Peter-Mark Verwoerd, Cloud Solutions Architect


Customers often ask us for guidance about how to build PCI DSS compliant environments on top of Google Cloud Platform. From our work in the field, we recently put together a handy-dandy tutorial to help them get started.

This is no small thing. Many businesses today have online storefronts, and the vast majority of those take credit cards. When you accept credit cards for your business, you have to make sure you do that securely — to ensure customer trust and security, to get paid and to meet the necessary regulations, namely PCI DSS.

The PCI DSS, created by the PCI Security Standards Council, is an information security standard created by the major credit card companies; as such, any business that takes Visa, MasterCard, Discover, American Express or JCB is expected to be PCI DSS compliant, and can be fined or penalized if it is not.

Creating and managing a compliant PCI DSS environment can be a non-trivial task. Thankfully, if you’re on Cloud Platform, managed services such as Stackdriver Monitoring, Stackdriver Logging, and Google BigQuery can help. Our solution, for example, includes these basic components:

• A lightweight Google Compute Engine front-end application that accepts credit card information and sends it to an external payment processor. Importantly, that information is never recorded, it's only transmitted.
• An external payment processor that charges the credit card if it's accepted or rejects it if it’s not, and notifies your application of the result. Since this is just a notification to your application, no credit card data is transmitted or recorded from the payment processor.
• Stackdriver Logging, which logs the actions of every application and server via Squid Proxy which restricts the event traffic and sends them to Stackdriver Monitoring, which monitors the events
• BigQuery, which can be used to analyze the logs, run ad-hoc audit queries and create reports.

(click to enlarge)

For further details, check out the full solution for this design. We hope you'll find it useful, and we welcome and encourage your feedback. Comment here or reach out to @petermark on Twitter.

Posted by Martin Buhr, Product Manager


The infrastructure underpinning Google Cloud Platform has always been good, and it keeps getting better. We recently increased the limit on the number of attached persistent disk storage volumes per Google Compute Engine instance from 16 to as many as 128 volumes on our largest standard instance sizes (giving us three times the capacity of the competition). We also recently increased the total quantity of attached persistent disk storage per instances from 10TB to as much as 64TB on our largest instance sizes, and introduced the ability to resize persistent disk volumes with no downtime.

(click to enlarge)

These changes were enabled by Google’s continued innovations in data center infrastructure, in this case networking. Combined with Colossus, Google’s high-performance global storage fabric, we have greatly increased the size, quantity, and flexibility of network attached storage per instance without sacrificing persistent disk’s legendary rock solid performance and reliability.

This ties back to one of Google Cloud Platform’s core strengths: hosting Linux containers. We are committed to making GCP the best place on the web to run Docker-based workloads, building on over a decade of experience running all of Google on Linux containers. To help you realize the same benefits that we did, we created and open-sourced Kubernetes, a tool for creating and managing clusters of Docker containers, and launched Google Container Engine to provide a managed, hosted, Kubernetes-based service for Docker applications.

Red Hat is a large Kubernetes user and has adopted it across its product portfolio, including making it a core component of OpenShift, its Platform as a Service offering. Working with Red Hat late last year in preparation for offering OpenShift Dedicated on Google Cloud Platform, it became clear that we needed to increase the number of attached persistent disk volumes per instance to help Red Hat efficiently bin pack Kubernetes Pods (each of which may have one or more attached volumes) across their Google Cloud Platform infrastructure.

Being able to attach more persistent disk volumes per instance isn’t just useful to Red Hat. In addition to better support for Kubernetes and Docker, this feature can also be used in various scenarios that require a large number of disks. For example, you keep data from different web servers on separate volumes to insure data isolation when an instance is running multiple web servers.

To take advantage of this feature for your own Kubernetes clusters running on Google Compute Engine, simply set the "KUBE_MAX_PD_VOLS" environment variable to whatever you want the limit to be, based on the maximum number of volumes the nodes in your cluster support.

Posted by Todd Prives, Product Manager, Google Cloud Platform

We're at NAB this week and are thrilled to announce new Google Cloud Platform partners and product updates that underscore the growing role of cloud computing in the media and entertainment industry.

We’re collaborating with Autodesk to launch a new cloud-optimized rendering service called Autodesk Maya for Cloud Platform ZYNC Render. Autodesk software has been behind the past 21 Academy Award winners for Best Visual Effects and we’re bringing this capability to Google Cloud Platform.

The combined offering captures the aim of both companies to enable individual artists and developers to focus on content, abstracting away the nuances of managing infrastructure. Small teams of artists can tap world-class infrastructure to realize the creative vision of what was once limited to much larger studios.

Here’s how it works:

• Product teams at Google and Autodesk have developed a cloud-optimized version of Autodesk® Maya®, the popular tool for animation, modeling and rendering. Via a Maya plugin, 3D scenes are transferred to Google Compute Engine in the background, while the artist is working. Because Maya is capable of running on the artist’s workstation and also in the cloud on GCP, this allows artists to run massively parallel rendering far more efficiently than before — taking advantage of the scalability, performance, and price benefits of GCP
• Compared with the non-optimized version, Maya® customers see up to 10x improvement in file upload efficiency. This allows many rendering jobs to start instantaneously, cutting wait time and accelerating time to finished shot

We’re also excited to announce ZYNC Render support for Pixar’s historic RenderMan, with licensing fully built-in to the product. ZYNC users are now able to spin up 500 machines per account, scaling to 32,000 rendering cores, with new support for 64-core machines — making short work of ambitious rendering jobs.

In addition to our collaboration with Autodesk, we've made some big strides in our offerings for the media and entertainment industry. Here are some recent updates:

Cloud Vision API graduates to General Availability

The goal of Cloud Vision API is to provide vision capability to your applications in the same way that Google Photos does. It's a powerful tool for media and entertainment companies enabling you to classify images and analyze emotional facial attributes. To further improve our customer experience, Cloud Vision API is going into general availability today with new features:

• Logo Detection expanded to cover millions of logos
• Updated OCR for understanding more granular text like receipts

Cloud CDN graduates to Beta

We’re launching Cloud CDN Beta, allowing your media content to be pushed out to Google’s network edge and cached close to users. As always, data travels via Google’s network and reaches users who expect instantaneous access to images and live-stream video experiences. Cloud CDN is also fully integrated with Google’s global load balancing and enterprise-grade security to distribute media workloads anywhere they originate, so jobs never get bogged down.

Lytro chooses Cloud Platform and The Foundry

One question we see today’s top innovators ask continuously is: with massive cloud infrastructure at our fingertips, how can we make a major leap forward in the way things are done today? Lytro is one example. Its technology seems to defy the traditional physics of photography, capturing massive volumes of visual data with such fidelity that infinite creative choices abound: unprecedented control over focus, perspective, aperture and shutter angle, all in post-production. Lytro selected GCP and The Foundry to help power their amazing invention. Learn more by watching their video.

For more on what's possible with cloud-enabled media, visit cloud.google.com/solutions/media and zyncrender.com, or contact us to discuss how cloud can enable your creative workflows.

Google Consumer Surveys has launched an API, built on Google Cloud Platform, that lets you create and integrate surveys as part of any mobile or web app.

We’ve spoken to research and non-research companies who are really interested in bringing the power of our Consumer Surveys tool into their own applications, and with this launch it’s finally possible. We imagine many different use-cases to tap into the millions of respondents our platform can connect you with, across a dozen or more markets around the world.

Leading up to our launch we worked closely with a handful of trusted testers who provided valuable feedback. These have included proprietary solutions to manage studies conducted on the Google Consumer Surveys platform (Kantar), predictive analytics solutions (Predictvia), and solutions that help customers visualize their Google Consumer Surveys data (MarketSight). We look forward to working with developers to build unique solutions that empower individuals and businesses to make better data driven decisions.

If you’re an existing Google Consumer Surveys Enterprise customer on an invoicing contract, you can start using the API immediately. If you’re not an Enterprise customer but are interested in accessing the API, you can get in touch with us at [email protected]. You can read more about our API on our API documentation site.