Posted Alex Barrett, Editor, Google Cloud Platform Blog

Judging by the spate of Kubernetes-related activity these days, it seems like the community's fired up and putting the open-source cluster management system to work in all sorts of interesting ways.

In recent days, we’ve seen proposals for new Kubernetes-compliant storage systems (Torus, by CoreOS, and KubeFuse by OpenCredo); new approaches to Kubernetes security (Hypernetes from HyperHQ); and even an enhanced version of Kubernetes inspired by its maker’s experience in the supercomputing market (NavOps Command, by Univa). And Google’s own Brendan Burns introduced ksql, which allows users to query Kubernetes objects using SQL.

What’s all the excitement about? In a nutshell, Chris Kleban argues that what Kubernetes (and Docker) bring to the table are “anywhere cloud services.” “Docker enables us to easily build, ship and run software by packaging it up in a way that will run on a wide range of systems,” he writes. “But, that isn't enough. We need a way to get that software installed, working and highly available. We need something like Kubernetes.”

Enterprises seem to be getting the memo too. Samsung SDS Research America told Timothy Prickett Morgan at the TheNextPlatform about how it’s putting Kubernetes through the paces, while Beth Pariseau of SearchITOperations.com details how Concur and Barkly Protects Inc. are using Kubernetes to manage applications running in Amazon Web Services and on-premises.

Of course, we recommend you run on Google Cloud Platform with Google Container Engine (GKE). As of last week, GKE now supports node pools, which make it possible to run nodes of different configurations within the same cluster.

If that’s still not enough Kubernetes for you, Google developer advocate Sandeep Dinesh writes about using kube-ui and Weave Scope with Google Container Engine, while Ian Lewis tackles the topic of using Kubernetes Health Checks.

Be sure to stop by soon for a piece of Kubernetes cake when the project turns two in July!

Posted by Alex Barrett, Editor, Google Cloud Platform Blog

Migrating to Google Cloud Platform isn’t hard per se, but there’s a lot to think about. And thankfully, it doesn’t need to happen all at once.

In his presentation, “Getting to the Cloud with Google Cloud Platform: A Sequential Approach” Google Cloud Solutions Architect Peter-Mark Verwoerd lays out a five-step plan to migrate workloads from an on-premises environment to GCP. It’s worth watching the webcast in its entirety, but until then, here are the high notes.

Phase One: Assess

Before you move a single bit, take stock of your applications and how suitable they are for the cloud. Things to think about include (but are not limited to) hardware and performance requirements, users, licensing, compliance needs and application dependencies.

Generally speaking, apps fall into one of three buckets: Easy to Move, Hard to Move, and Can’t Move. In our experience, applications that most often fall into the Easy to Move bucket are greenfield apps (no surprise there), test and dev and Q&A. Internal web apps and batch processing applications are also good cloud candidates, because they can scale horizontally rather than vertically.

Phase Two: Pilot

This is the point where you take one or two applications, and try moving them. Learn about Cloud Platform and its design patterns, take the time to validate performance, consider your licensing options and establish how to perform a rollback. Don’t skip this step, and don’t be tempted to try and migrate too many apps all at once!

Phase Three: Move Data

Some people will tell you to move your applications first, then move your data, but we beg to differ. Most applications have a lot of data with a lot of dependencies. Properly moving data to the cloud sets the stage for a successful application migration later on.

This is also the time to consider your various cloud storage options — regular Google Cloud Storage or Nearline? Local SSDs or persistent disks? Google Cloud SQL, Datastore or Bigtable? You should also think about how you’re going to move all that data — via batch data transfers, offline disk imports, with database dumps, or streaming to persistent disks? There are lots of things to consider here.

Phase Four: Move Applications

Now that your data is in the cloud, you’re ready to move the actual apps. Here too, there are decisions to make. We recommend keeping things simple, and doing the minimum necessary to get the application up and running in the cloud, for example doing a straight lift-and-shift. Or perhaps you can get an app into the cloud by way of backing it up there? That way, in the event of an outage, there’s a full copy of your environment in GCP waiting to take over.

Phase Five: Optimize

This is where the fun begins. Once an application and its data have been migrated to GCP, you can start thinking about all the cool ways to make it better. For example, this might be a time to add redundancy in the form of availability zones, elasticity with autoscaling groups, or enhanced monitoring with Stackdriver. You might want to offload static assets off of your application tier into Cloud Storage, or decouple tiers by using Pub/Sub. Google’s Deployment Manager can make it easier to launch and scale new instances, and copying your configuration into a second region can insulate you from a regional outage.

See? That wasn’t so bad. In the meantime, if you’d like to turbocharge your VM migration process, we also have systems integration partners that are cloud migration and GCP experts and would be happy to assist you. You can find more resources on migrating to GCP, including a list of certifed partners, at https://cloud.google.com/migrate/. Here’s the webinar once again, for a more detailed step-by-step guide.

Posted by Jo Maitland, Managing Editor, Google Cloud Platform Blog

In-between daydreaming about virtual reality and saying Allo to Duo, there were some fantastic Google Cloud Platform sessions and demos at Google I/O this year. In case you weren’t able to attend the show, here are the recordings.

In One Lap Around Google Cloud Platform, senior developer advocate Mandy Waite, and Brad Abrams walk through the process of building a Node.js backend for an iOS and Android based game. Easily the best bit of the session is a demo that uses Kubernetes and GCP Container Registry to deploy a Docker container to App Engine Flexible Environment, Google Container Engine and an AWS EC2 instance. It’s a simple demo of portability across clouds, a key differentiator for GCP.

Speaking of multi-cloud environments, developer advocate Aja Hammerly presented a great session on Stackdriver, called Just Enough Stackdriver To Sleep Well At Night, which will be music to the ears for any of you who have to carry a pager to support your site or application. In a nutshell, Aja shows how Stackdriver unifies a bunch of different monitoring and management tools into a single solution.

If you’ve made the leap to containers or are thinking about it, you’ll want to check Carter Morgan's session: Best practices for orchestrating the cloud with Kubernetes. This session includes the basics of modern day applications and how containers work. It also covers packaging and distributing apps using Docker and how to up your game by running applications on Kubernetes.

Did you know Kubernetes has seen 5,000 commits and over 50% from unique contributors since January 2015?

Next, IoT ideas are a dime a dozen, but bringing them to life is another story. In Making sense of IoT data with the cloud, developer advocate Ian Lewis shows how you can manage a large number of devices on GCP and how to ingest, store and analyze the data from those devices.

In Supercharging Firebase with Google Cloud Platform developer advocates Sandeep Dinesh and Bret McGowen use Firebase to build a real-time game that interacts with virtual machines, big data and machine learning APIs on GCP. The coolest part of the demo involves the audience in the room and on the livestream interacting with the game via Speech API, all yelling instructions at the same time to move a dot through a maze. The hallmark of Firebase — real-time data synchronization across connected devices in milliseconds — is on display here and fun to see. For more Firebase tips and tricks check out Creating interactive multiplayer experiences with Firebase, from developer advocate Mark Mandel.

Switching gears to big data and the upcoming U.S. presidential election, developer advocate Felipe Hoffa and software engineer Jordan Tigani, demo the power of Google BigQuery to uncover some intriguing campaign insights in Election 2016: The Big Data Showdown. You'll learn which candidate is spending the most money and how efficient that spending is relative to their mentions on TV, by mashing together various different public datasets in BigQuery. Felipe and Jordan do a nice job showing us how BigQuery can separate the signal from the noise to figure out what it all means.

Figuring out the right storage for each application in your business can be a daunting task on the cloud. Dominic Preuss, group product manager, explains how in Scaling your data from concept to petabytes.
And of course, no event that includes Cloud Platform is complete without demos from developer advocates Kaz Sato on How to build a smart RasPi bot with Cloud Vision and Speech API, and another crowd-pleaser, Google Cloud Spin: Stopping time with the power of Cloud, from Francesc Campoy Flores.

To find more tutorials, talks and demos on GCP beyond the sessions at I/O this year, check out our GCP YouTube channel and weekly podcast, and follow @GoogleCloud on Twitter for all the latest news and product announcements from the Cloud Platform team.

Posted by Alex Barrett, Editor, Google Cloud Platform Blog

Is there a limit to what you can do with machine learning? Doesn’t seem like it.

At Moogfest last week, Google researchers presented about Project Magenta, which uses TensorFlow, the open-source machine learning library that we developed, to see if computers can create original pieces of art and music.

Researchers have also shown that they can use TensorFlow to train systems to imitate the grand masters. With this implementation of neural style in TensorFlow, it becomes easy to render an image that looks like it was created by Vincent Van Gogh or Pablo Picasso — or a combination thereof. Take this image of Frank Gehry’s Stata Center in winter,
add style inputs from Van Gogh’s Starry Night and a Picasso Dora Maar:
and end up with:
Voila! A Picasso-esque Van Gogh for 21st century!

(The code for neural style was first posted to GitHub last December, but the author continues to update it and welcomes pull requests.)

Or maybe fine art isn’t your thing. This week, we also saw how to use TensorFlow to solve trivial programming problems, forecast demand — even predict the elections.

Because TensorFlow is open source, anyone can use it, on the platform of their choice. But it’s worth mentioning that running machine learning on Google Cloud Platform works especially well. You can learn more about GCP’s machine learning capabilities here. And if you’re doing something awesome with machine learning on GCP, we’d love to hear about it — just tweet us at @googlecloud.

Posted by Fabio Yeon, Software Engineer, Google Cloud Platform


Google Container Engine (GKE) aims to be the best place to set up and manage your Kubernetes clusters. When creating a cluster, users have always been able to select options like the nodes’ machine type, disk size, etc. but that applied to all the nodes, making the cluster homogenous. Until now, it was very difficult to have a cluster with a heterogenous machine configuration.

That’s where node pools come in, a new feature in Google Container Engine that's now generally available. A node pool is simply a collection, or “pool,” of machines with the same configuration. Now instead of a uniform cluster where all the nodes are the same, you can have multiple node pools that better suit your needs. Imagine you created a cluster composed of n1-standard-2 machines, and realize that you need more CPU. You can now easily add a node pool to your existing cluster composed of n1-standard-4 (or bigger) machines.

All this happens through the new “node-pools” commands available via the gcloud command line tool. Let’s take a deeper look at using this new feature.

Creating your cluster

A node pool must belong to a cluster and all clusters have a default node pool named “default-pool”. So, let’s create a new cluster (we assume you’ve set the project and zone defaults in gcloud):

> gcloud container clusters create work
NAME     ZONE           MASTER_VERSION  MASTER_IP        MACHINE_TYPE   
NODE_VERSION  NUM_NODES  STATUS
work     us-central1-f  1.2.3           123.456.789.xxx  n1-standard-1  
1.2.3         3          RUNNING

Like before, you can still specify some node configuration options, like “--machine-type” to specify a machine type, or “--num-nodes” to set the initial number of nodes.

Creating a new node pool

Once the cluster has been created, you can see its node pools with the new “node-pools” top level object (Note: You may need to upgrade your gcloud commands via “gcloud components update” to use these new options.).

> gcloud container node-pools list --cluster=work
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.2.3

Notice that you must now specify a new parameter, “--cluster”. Recall that node pools belong to a cluster, so you must specify the cluster with which to use node-pools commands. You can also set it as the default in config by calling:

> gcloud config set container/cluster 

Also, if you have an existing cluster on GKE, your clusters will have been automatically migrated to “default-pool,” with the original cluster node configuration.

Let’s create a new node pool on our “work” with a custom machine type of 2 CPUs and 12 GB of RAM:

> gcloud container node-pools create high-mem --cluster=work 
--machine-type=custom-2-12288 --disk-size=200 --num-nodes=4

This creates a new node pool with 4 nodes, using custom machine VMs and 200 GB boot disks. Now, when you list your node pools, you get:

> gcloud container node-pools list --cluster=work
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.2.3
high-mem      custom-2-12288 200           1.2.3

And if you list the nodes in kubectl:

> kubectl get nodes
NAME                                 STATUS    AGE
gke-work-high-mem-d8e4e9a4-xzdy       Ready     2m
gke-work-high-mem-d8e4e9a4-4dfc       Ready     2m
gke-work-high-mem-d8e4e9a4-bv3d       Ready     2m
gke-work-high-mem-d8e4e9a4-5312       Ready     2m
gke-work-default-pool-9356555a-uliq   Ready     1d

With Kubernetes 1.2, the nodes on each node pool are also automatically assigned the node label, “cloud.google.com/gke-nodepool=”. With node labels, it’s possible to have heterogeneous nodes within your cluster, and schedule your pods into the specific nodes that meet their needs. Perhaps a set of pods need a lot of memory — allocate a high-mem node pool and schedule them there. Or perhaps they need more local disk space — assign them to a node pool with a lot of local storage capacity. More configuration options for nodes are being considered.

More fun with node pools

There are also other, more advanced scenarios for node pools. Suppose you want to upgrade the nodes in your cluster to the latest Kubernetes release, but need finer grained control of the transition (e.g., to perform A/B testing, or to migrate the pods slowly). When a new release of Kubernetes is available on GKE, simply create a new node pool; all node pools have the same version as the cluster master, which will be automatically updated to the latest Kubernetes release. Here’s how to create a new node pool with the appropriate version:

> gcloud container node-pools create my-1-2-4-pool --cluster=work 
--num-nodes=3 --machine-type=n1-standard-4

> gcloud container node-pools list --cluster=work
NAME           MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool   n1-standard-1  100           1.2.3
high-mem       custom-2-12288 200           1.2.3
my-1-2-4-pool  n1-standard-4  100           1.2.4

You can now go to “kubectl” and update your replication controller to schedule your pods with the label selector “cloud.google.com/gke-nodepool=my-1-2-4-pool”. Your pods will then be rescheduled from the old nodes to the new pool nodes. After the verifications are complete, continue the transition with other pods, until all of the old nodes are effectively empty. You can then delete your original node pool:

> gcloud container node-pools delete default-pool --cluster=work

> gcloud container node-pools list --cluster=work
NAME             MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
high-mem         custom-2-12288 200           1.2.3
My-1-2-4-pool    n1-standard-4  100           1.2.4

And voila, all of your pods are now running on nodes running the latest version of Kubernetes!

Conclusion

The new node pools feature in GKE enables more powerful and flexible scenarios for your Kubernetes clusters. As always, we’d love to hear your feedback and help guide us on what you’d like to see in the product.

Posted by Terrence Ryan, Developer Advocate

Last week at OSCON, a couple of us Google Cloud Platform developer advocates participated in a hackathon with the Kubernetes team and the Cloud Native Computing Foundation as part of the OSCON Contribute series. The idea behind these hackathons is to help people get involved with open source, to grow from users to contributors. Every open source project has its own culture around submissions, and can be daunting to people on the outside. Having people who want to contribute in the same room as the people who know the process can get new contributors up and running quickly.

John Mulhausen (standing left) from the Technical Writing team and Jeff Mendoza (standing right) from Developer Programs Engineering roam the tables and help Kubernetes hackers get started.

We had a diverse crowd of people from all around the Kubernetes community. One group was working on building drivers to connect EMC storage as a persistent storage solution for Kubernetes clusters. Another contributor helped improve documentation by submitting diagrams that illustrate complex configurations. Mike from Wisconsin participated because “I just love this project. I wanted to see if I could do something to give back.” We liked that attitude so a few of us Googlers skipped out on the event lunch and took Mike out for high-end hot dogs at Frank Restaurant.

Around 75 people showed up to the event, and by the end of the day we had seven pull requests from four new contributors for Kubernetes documentation, and a couple of the 43 pull requests the main Kubernetes project fielded were from our efforts. We’re happy to have the contributions, and even more excited for the new contributors, who have gotten over the hurdle of putting in their first pull request. They're that much closer to their second and third contributions than they were before this event, and are now valued members of the Kubernetes contributor community.

We also weren’t above bribing people. We gave out lots of great new limited-edition “Kollaborator” t-shirts, Cloud Platform credits, and free, temporary accounts for non-GCP users. If you have to run some servers to test out your Kubernetes contributions, we want to foot the bill!

Just because the hackathon is over, it doesn’t mean you can’t still contribute. You can find out more about the Kubernetes Project at our Getting Started pages. Then, visit the Kubernetes project on Github, click on the “Fork” button and start messing around with the source. If you want to try your hand at a few areas where we need help, check out the issues on Github marked Help Wanted. Our documentation team is so keen on getting contributions that they've even put up bounties on particular issues. Every little bit counts!

Posted by Alex Barrett, Editor, Google Cloud Platform Blog


If you’re looking for a fun way to get up-to-speed on Kubernetes and microservices without leaving the comfort of your desk, look no further. The new online course “Scalable Microservices with Kubernetes” on Udacity is designed for developers and systems administrators that want to hone their automation and distributed systems management skills, and is led by Google Cloud Platform’s very own Kelsey Hightower and Carter Morgan. There’s also a guest appearance from Battery Ventures Technology Fellow and DevOps legend Adrian Cockcroft!

This intermediate-level class consists of four lessons:

• Introduction to Microservices
• Building Containers with Docker
• Kubernetes 
• Deploying Microservices (from image to running service)

To excel at the course, you should be fluent in at least one programming language, be familiar with general virtualization concepts and know your way around a Linux command line. Udacity estimates that students can complete the course in about one month.

Successful completion of the course counts towards Udacity’s upcoming DevOps Engineer Nanodegree — one of several Nanodegrees co-developed by Google. Best of all, the course is free!
Here’s where to sign up.

Posted by Alex Barrett, Editor, Google Cloud Platform Blog

With all eyes trained on Google I/O, our annual developer show, Google Cloud Platform took the opportunity to share some of our secret sauce on why GCP is the best possible cloud on which to build and run enterprise applications.

From the stealthy Tensor Processing Unit (TPU), a custom chip to power machine learning apps, to new Firebase integrations with GCP, the industry default mobile backend-as-service, to new APIs for Sheets and Slides — it’s hard to keep your hands off this stuff if you have an inclination to build the next big thing in the enterprise.

In other news, did you know that GCP now recommends a container-optimized OS image based on Chromium OS — the open-source version of Chrome OS? With Container-VM Image, GCP users gain better control over build management, security and compliance and customizations for services such as Google Compute Engine’s metadata framework and image packages, the Docker runtime and Kubernetes. Currently in beta, here’s some helpful documentation on how to get started.
Running containerized workloads may be the next big thing, but GCP customers still spend a lot of time and energy sending out email. A lot of email. To recipients all over the world. For those users with European customers, email marketing firm Mailjet announced this week that several of its services can be accessed directly from Google Compute Engine and Google App Engine. Furthermore, thanks to servers located on the Old Continent, it's fully compliant with U.S. as well as European Union privacy regulations. Whether you pronounce it /ˈpɹɪv.ə.si/ or ˈpɹaɪ.və.si/, Mailjet and GCP have got you covered.

Indeed, with cloud services, location is almost as important as the service itself. Data Center Knowledge gives us a great Q&A with Google vice president of datacenter operations Joe Kava in which he talks about GCP’s plans to launch the new regions in the coming months, including Tokyo, and why in the cloud, it’s so important to build data centers in major metropolitan areas.
Check back next week for our highlights reel of the best Cloud Platform sessions and demos from Google I/O.

Posted by James Tamplin and Francis Ma, Firebase Product Leads

In the short time since Firebase joined Google, the passionate community of developers using the backend-as-a-service to handle the heavy lifting of building an app has grown from 110,000 to over 470,000 developers around the world.

In that same span, Firebase has come to rely on Google Cloud Platform, leaning on GCP for core infrastructure as well as value-added services. For example, GCP figures prominently in several of the new Firebase features that we’re announcing at Google I/O 2016 today.

One of the most requested features by Firebase developers is the ability to store images, videos and other large files. The new Firebase Storage is powered by Google Cloud Storage, giving it massive scalability and allowing stored files to be easily accessed by other projects running on Google Cloud Platform.

Firebase now uses the same underlying account system as GCP, which means you can use any GCP product with your Firebase app. For example, you can export raw analytics data from the new Firebase Analytics to Google BigQuery to help you surface advanced insights about your application and users.

Going forward, we’ll continue to build out integrations between Firebase and Google Cloud Platform, giving you the functionality of a full public cloud as you add to your mobile application portfolio.

To learn more about Firebase, visit our new site. For all the new features we’re announcing at Google I/O today, click on over to the Firebase blog. We’re working quickly to close gaps, and we’d love to hear your feedback so we can improve. You can help by requesting a feature.

Posted by Norm Jouppi, Distinguished Hardware Engineer, Google


Machine learning provides the underlying oomph to many of Google’s most-loved applications. In fact, more than 100 teams are currently using machine learning at Google today, from Street View, to Inbox Smart Reply, to voice search.

But one thing we know to be true at Google: great software shines brightest with great hardware underneath. That’s why we started a stealthy project at Google several years ago to see what we could accomplish with our own custom accelerators for machine learning applications.

The result is called a Tensor Processing Unit (TPU), a custom ASIC we built specifically for machine learning — and tailored for TensorFlow. We’ve been running TPUs inside our data centers for more than a year, and have found them to deliver an order of magnitude better-optimized performance per watt for machine learning. This is roughly equivalent to fast-forwarding technology about seven years into the future (three generations of Moore’s Law).

TPU is tailored to machine learning applications, allowing the chip to be more tolerant of reduced computational precision, which means it requires fewer transistors per operation. Because of this, we can squeeze more operations per second into the silicon, use more sophisticated and powerful machine learning models and apply these models more quickly, so users get more intelligent results more rapidly. A board with a TPU fits into a hard disk drive slot in our data center racks.

Tensor Processing Unit board

TPU is an example of how fast we turn research into practice — from first tested silicon, the team had them up and running applications at speed in our data centers within 22 days.

TPUs already power many applications at Google, including RankBrain, used to improve the relevancy of search results and Street View, to improve the accuracy and quality of our maps and navigation. AlphaGo was powered by TPUs in the matches against Go world champion, Lee Sedol, enabling it to "think" much faster and look farther ahead between moves.

Server racks with TPUs used in the AlphaGo matches with Lee Sedol

Our goal is to lead the industry on machine learning and make that innovation available to our customers. Building TPUs into our infrastructure stack will allow us to bring the power of Google to developers across software like TensorFlow and Cloud Machine Learning with advanced acceleration capabilities. Machine Learning is transforming how developers build intelligent applications that benefit customers and consumers, and we're excited to see the possibilities come to life.